tag:blogger.com,1999:blog-79629345334890045822024-03-18T10:42:49.547-07:00Steve HardyOpen-source, Linux, OpenStack, Heat, programmingSteve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-7962934533489004582.post-19826096474169952132018-06-04T10:09:00.001-07:002018-06-04T10:09:25.305-07:00TripleO Containerized deployments, debugging basics<h3>
Containerized deployments, debugging basics</h3>
Since the Pike release, TripleO has supported deployments with OpenStack services running in containers. Currently we use docker to run images based on those maintained by the <a href="https://docs.openstack.org/kolla/latest/">Kolla project</a>.<br />
<br />
We already have some <a href="https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/tips_tricks.html">tips and tricks</a> for container deployment debugging in <a href="https://docs.openstack.org/tripleo-docs/latest/">tripleo-docs,</a> but below are some more notes on my typical debug workflows.<br />
<br />
<h3>
Config generation debugging overview</h3>
In the TripleO container architecture, we still
use Puppet to generate configuration files and do some bootstrapping,
but it is run (inside a container) via a script <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/docker/docker-puppet.py">docker-puppet.py</a><br />
<br />
The config generation usage happens at the start of the deployment (step 1) and the configuration files are generated for all services (regardless of which step they are started in).<br />
<br />
The input file used is /var/lib/docker-puppet/docker-puppet.json, but you can also filter this (e.g via cut/paste or jq as shown below) to enable debugging for specific services - this is helpful when you need to iterate on debugging a config generation issue for just one service.<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[root@overcloud-controller-0 docker-puppet]# jq '[.[]|select(.config_volume | contains("heat"))]' /var/lib/docker-puppet/docker-puppet.json | tee /tmp/heat_docker_puppet.json
{
"puppet_tags": "heat_config,file,concat,file_line",
"config_volume": "heat_api",
"step_config": "include ::tripleo::profile::base::heat::api\n",
"config_image": "192.168.24.1:8787/tripleomaster/centos-binary-heat-api:current-tripleo"
}
{
"puppet_tags": "heat_config,file,concat,file_line",
"config_volume": "heat_api_cfn",
"step_config": "include ::tripleo::profile::base::heat::api_cfn\n",
"config_image": "192.168.24.1:8787/tripleomaster/centos-binary-heat-api-cfn:current-tripleo"
}
{
"puppet_tags": "heat_config,file,concat,file_line",
"config_volume": "heat",
"step_config": "include ::tripleo::profile::base::heat::engine\n\ninclude ::tripleo::profile::base::database::mysql::client",
"config_image": "192.168.24.1:8787/tripleomaster/centos-binary-heat-api:current-tripleo"
}
</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
Then we can run the config generation, if necessary changing the tags (or puppet modules, which are consumed from the host filesystem e.g /etc/puppet/modules) until the desired output is achieved:<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">
[root@overcloud-controller-0 docker-puppet]# export NET_HOST='true'
[root@overcloud-controller-0 docker-puppet]# export DEBUG='true'
[root@overcloud-controller-0 docker-puppet]# export PROCESS_COUNT=1
[root@overcloud-controller-0 docker-puppet]# export CONFIG=/tmp/heat_docker_puppet.json
[root@overcloud-controller-0 docker-puppet]# python /var/lib/docker-puppet/docker-puppet.py2018-02-09 16:13:16,978 INFO: 102305 -- Running docker-puppet
2018-02-09 16:13:16,978 DEBUG: 102305 -- CONFIG: /tmp/heat_docker_puppet.json
2018-02-09 16:13:16,978 DEBUG: 102305 -- config_volume heat_api
2018-02-09 16:13:16,978 DEBUG: 102305 -- puppet_tags heat_config,file,concat,file_line
2018-02-09 16:13:16,978 DEBUG: 102305 -- manifest include ::tripleo::profile::base::heat::api
2018-02-09 16:13:16,978 DEBUG: 102305 -- config_image 192.168.24.1:8787/tripleomaster/centos-binary-heat-api:current-tripleo
...
</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
When the config generation is completed, configuration files are written out to /var/lib/config-data/heat.<br />
<br />
We then compare timestamps against the /var/lib/config-data/heat/heat.*origin_of_time file (touched for each service before we run the config-generating containers), so that only those files modified or created by puppet are copied to /var/lib/config-data/puppet-generated/heat.<br />
<br />
Note that we also calculate a checksum for each service (see /var/lib/config-data/puppet-generated/*.md5sum), which means we can detect when the configuration changes - when this happens we need <a href="https://github.com/openstack/paunch">paunch</a> to restart the containers, even though the image did not change.<br />
<br />
This checksum is added to the /var/lib/tripleo-config/hashed-docker-container-startup-config-step_*.json files by docker-puppet.py, and these files are later used by paunch to decide if a container should be restarted (see below).
<br />
<h3>
</h3>
<h3>
Runtime debugging, paunch 101</h3>
Paunch is a tool that orchestrates launching containers for each step, and performing any bootstrapping tasks not handled via docker-puppet.py.<br />
<br />
It accepts a json format, which are the<i> /var/lib/tripleo-config/docker-container-startup-config-step_*.json</i> files that are created based on the enabled services (the content is directly derived from the service templates in tripleo-heat-templates)<br />
<br />
These json files are then modified via docker-puppet.py (as mentioned above) to add a <i>TRIPLEO_CONFIG_HASH</i> value to the container environment - these modified files are written with a different name, see <i>/var/lib/tripleo-config/hashed-docker-container-startup-config-step_*.json</i><br />
<br />
Note this environment variable isn't used by the container directly, it is used as a salt to trigger restarting containers when the configuration files in the mounted config volumes have changed.<br />
<br />
As in the docker-puppet case it's possible to filter the json file with jq and debug e.g mounted volumes or other configuration changes directly.<br />
<br />
It's also possible to test configuration changes by manually modifying /var/lib/config-data/puppet-generated/ then either restarting the container via docker restart, or by modifying <i>TRIPLEO_CONFIG_HASH</i> then re-running paunch. <br />
<br />
<b>Note</b> paunch will kill any containers tagged for a particular step e.g the <i>--config-id tripleo_step4 --managed-by tripleo-Controller</i> means all containers started during this step for any previous paunch apply will be killed<b> </b>if they are removed from your json during testing. This is a feature which enables changes to the enabled services on update to your overcloud but it's worth bearing in mind when testing as described here.<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">
[root@overcloud-controller-0]# cd /var/lib/tripleo-config/
[root@overcloud-controller-0 tripleo-config]# jq '{"heat_engine": .heat_engine}' hashed-docker-container-startup-config-step_4.json | tee /tmp/heat_startup_config.json
{
"heat_engine": {
"healthcheck": {
"test": "/openstack/healthcheck"
},
"image": "192.168.24.1:8787/tripleomaster/centos-binary-heat-engine:current-tripleo",
"environment": [
"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
"TRIPLEO_CONFIG_HASH=14617e6728f5f919b16c74f1e98d0264"
],
"volumes": [
"/etc/hosts:/etc/hosts:ro",
"/etc/localtime:/etc/localtime:ro",
"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
"/dev/log:/dev/log",
"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
"/etc/puppet:/etc/puppet:ro",
"/var/log/containers/heat:/var/log/heat",
"/var/lib/kolla/config_files/heat_engine.json:/var/lib/kolla/config_files/config.json:ro",
"/var/lib/config-data/puppet-generated/heat/:/var/lib/kolla/config_files/src:ro"
],
"net": "host",
"privileged": false,
"restart": "always"
}
}
[root@overcloud-controller-0 tripleo-config]# paunch --debug apply --file /tmp/heat_startup_config.json --config-id tripleo_step4 --managed-by tripleo-Controller
stdout: dd60546daddd06753da445fd973e52411d0a9031c8758f4bebc6e094823a8b45
stderr:
[root@overcloud-controller-0 tripleo-config]# docker ps | grep heat
dd60546daddd 192.168.24.1:8787/tripleomaster/centos-binary-heat-engine:current-tripleo "kolla_start" 9 seconds ago Up 9 seconds (health: starting) heat_engine
</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<h3>
</h3>
<h3>
Containerized services, logging</h3>
There are a couple of ways to access the container logs:<br />
<br />
<ul>
<li>On the host filesystem, the container logs are persisted under /var/log/containers/<service></li>
<li>docker logs <container id or name></li>
</ul>
It is also often useful to use docker inspect <container id or name> to verify the container configuration, e.g the image in use and the mounted volumes etc. <br />
<h3>
</h3>
<h3>
Debugging containers directly</h3>
Sometimes logs are not enough to debug problems, and in this case you must interact with the container directly to diagnose the issue.<br />
<br />
When a container is not restarting, you can attach a shell to the running container via docker exec:<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">
[root@openstack-controller-0 ~]# docker exec -ti heat_engine /bin/bash
()[heat@openstack-controller-0 /]$ ps ax
PID TTY STAT TIME COMMAND
1 ? Ss 0:00 /usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start
5 ? Ss 1:50 /usr/bin/python /usr/bin/heat-engine --config-file /usr/share/heat/heat-dist.conf --config-file /etc/heat/heat
25 ? S 3:05 /usr/bin/python /usr/bin/heat-engine --config-file /usr/share/heat/heat-dist.conf --config-file /etc/heat/heat
26 ? S 3:06 /usr/bin/python /usr/bin/heat-engine --config-file /usr/share/heat/heat-dist.conf --config-file /etc/heat/heat
27 ? S 3:06 /usr/bin/python /usr/bin/heat-engine --config-file /usr/share/heat/heat-dist.conf --config-file /etc/heat/heat
28 ? S 3:05 /usr/bin/python /usr/bin/heat-engine --config-file /usr/share/heat/heat-dist.conf --config-file /etc/heat/heat
2936 ? Ss 0:00 /bin/bash
2946 ? R+ 0:00 ps ax
</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
That's all for today, for more information please refer to <a href="https://docs.openstack.org/tripleo-docs/latest/">tripleo-docs,</a>, or feel free to ask questions in #tripleo on Freenode!Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com21tag:blogger.com,1999:blog-7962934533489004582.post-8260983384240032972018-02-09T09:04:00.000-08:002018-02-09T09:04:23.972-08:00Debugging TripleO revisited - Heat, Ansible & PuppetSome time ago I wrote a post about <a href="https://hardysteven.blogspot.co.uk/2015/04/debugging-tripleo-heat-templates.html">debugging TripleO heat templates</a>, which contained some details of possible debug workflows when TripleO deployments fail.<br />
<br />
In recent releases (since the Pike release) we've made some major changes to the TripleO architecture - we makes more use of Ansible "under the hood", and we now support deploying containerized environments. I described some of these architectural changes in a <a href="https://www.openstack.org/videos/sydney-2017/deploying-openstack-at-scale-with-tripleo-ansible-and-containers">talk at the recent OpenStack Summit in Sydney</a>.<br />
<br />
In this post I'd like to provide a refreshed tutorial on typical debug workflow, primarily focussing on the configuration phase of a typical TripleO deployment, and with particular focus on interfaces which have changed or are new since my <a href="https://hardysteven.blogspot.co.uk/2015/04/debugging-tripleo-heat-templates.html">original debugging post</a>. <br />
<br />
We'll start by looking at the deploy workflow as a whole, some heat interfaces for diagnosing the nature of the failure, then we'll at how to debug directly via Ansible and Puppet. In a future post I'll also cover the basics of debugging containerized deployments.<br />
<br />
<h3>
The TripleO deploy workflow, overview</h3>
<h3>
<span id="goog_178782323"></span><span id="goog_178782324"></span></h3>
A typical TripleO deployment consists of several discrete phases, which are run in order:<br />
<br />
<h3>
Provisioning of the nodes</h3>
<br />
<ol>
<li>A "plan" is created (heat templates and other files are uploaded to Swift running on the undercloud</li>
<li>Some validation checks are performed by Mistral/Heat then a Heat stack create is started (by Mistral on the undercloud)</li>
<li>Heat creates some groups of nodes (one group per TripleO role e.g "Controller"), which results in API calls to Nova</li>
<li>Nova makes scheduling/placement decisions based on your flavors (which can be different per role), and calls Ironic to provision the baremetal nodes</li>
<li>The nodes are provisioned by Ironic</li>
</ol>
<br />
This first phase is the provisioning workflow, after that is complete and the nodes are reported ACTIVE by nova (e.g the nodes are provisioned with an OS and running).<br />
<br />
<h3>
Host preparation </h3>
The next step is to configure the nodes in preparation for starting the services, which again has a specific workflow (some optional steps are omitted for clarity):<br />
<br />
<ol>
<li>The node networking is configured, via the os-net-config tool</li>
<li>We write hieradata for puppet to the node filesystem (under /etc/puppet/hieradata/*)</li>
<li>We write some data files to the node filesystem (a puppet manifest for baremetal configuration, and some json files that are used for container configuration)</li>
</ol>
<br />
<h3>
Service deployment, step-by-step configuration </h3>
The final step is to deploy the services, either on the baremetal host or in containers, this consists of several tasks run in a specific order:<br />
<br />
<ol>
<li>We run puppet on the baremetal host (even in the containerized architecture this is still needed, e.g to configure the docker daemon and a few other things)</li>
<li>We run "docker-puppet.py" to generate the configuration files for each enabled service (this only happens once, on step 1, for all services)</li>
<li>We start any containers enabled for this step via the "paunch" tool, which translates some json files into running docker containers, and optionally does some bootstrapping tasks.</li>
<li>We run docker-puppet.py again (with a different configuration, only on one node the "bootstrap host"), this does some bootstrap tasks that are performed via puppet, such as creating keystone users and endpoints after starting the service.</li>
</ol>
<br />
Note that these steps are performed repeatedly with an incrementing step value (e.g step 1, 2, 3, 4, and 5), with the exception of the "docker-puppet.py" config generation which we only need to do once (we just generate the configs for all services regardless of which step they get started in).<br />
<br />
Below is a diagram which illustrates this step-by-step deployment workflow:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2XlIT8Ovj18paZfAkTaUcljuMwBqtTL-yjojXpYPvhsZlpZIn7xMw8sO119MzNoKgzZ2KrWP7aZ_cPIDYnae3AfAGVIxpcNvDoB53PK0qknSUTVrQWDhvJlFj1NOIHNUW7GirOzLcAQE/s1600/Deploy+Steps.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="357" data-original-width="960" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2XlIT8Ovj18paZfAkTaUcljuMwBqtTL-yjojXpYPvhsZlpZIn7xMw8sO119MzNoKgzZ2KrWP7aZ_cPIDYnae3AfAGVIxpcNvDoB53PK0qknSUTVrQWDhvJlFj1NOIHNUW7GirOzLcAQE/s640/Deploy+Steps.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">TripleO Service configuration workflow</td></tr>
</tbody></table>
<br />
The most common deployment failures occur during this service configuration phase of deployment, so the remainder of this post will primarily focus on debugging failures of the deployment steps.<br />
<h3>
</h3>
<h3>
Debugging first steps - what failed?</h3>
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">Heat Stack create failed.</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
Ok something failed during your TripleO deployment, it happens to all of us sometimes! The next step is to understand the root-cause. <br />
<br />
My starting point after this is always to run:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">openstack stack failures list --long <stackname></span><br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">(undercloud) [stack@undercloud ~]$ openstack stack failures list --long overcloud</span>
<span style="color: #888888;">overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:</span>
<span style="color: #888888;"> resource_type: OS::Heat::StructuredDeployment</span>
<span style="color: #888888;"> physical_resource_id: 421c7860-dd7d-47bd-9e12-de0008a4c106</span>
<span style="color: #888888;"> status: CREATE_FAILED</span>
<span style="color: #888888;"> status_reason: |</span>
<span style="color: #888888;"> Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2</span>
<span style="color: #888888;"> deploy_stdout: |</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> PLAY [localhost] ***************************************************************</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> ...</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> TASK [Run puppet host configuration for step 1] ********************************</span>
<span style="color: #888888;"> ok: [localhost]</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> TASK [debug] *******************************************************************</span>
<span style="color: #888888;"> fatal: [localhost]: FAILED! => {</span>
<span style="color: #888888;"> "changed": false, </span>
<span style="color: #888888;"> "failed_when_result": true, </span>
<span style="color: #888888;"> "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [</span>
<span style="color: #888888;"> "Debug: Runtime environment: puppet_version=4.8.2, ruby_version=2.0.0, run_mode=user, default_encoding=UTF-8", </span>
<span style="color: #888888;"> "Error: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'ugeas' at /etc/puppet/modules/tripleo/manifests/profile/base/docker.pp:181:5 on node overcloud-controller-0.localdomain"</span>
<span style="color: #888888;"> ]</span>
<span style="color: #888888;"> }</span>
<span style="color: #888888;"> to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/8dd0b23a-acb8-4e11-aef7-12ea1d4cf038_playbook.retry</span>
<span style="color: #888888;"> </span>
<span style="color: #888888;"> PLAY RECAP *********************************************************************</span>
<span style="color: #888888;"> localhost : ok=18 changed=12 unreachable=0 failed=1 </span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
We can tell several things from the output (which has been edited above for brevity), firstly the name of the failing resource<br />
<br />
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0</span></pre>
<ul>
<li>The error was on one of the Controllers (ControllerDeployment)</li>
<li>The deployment failed during the per-step service configuration phase (the AllNodesDeploySteps part tells us this)</li>
<li>The failure was during the first step (Step1.0)</li>
</ul>
Then we see more clues in the deploy_stdout, ansible failed running the task which runs puppet on the host, it looks like a problem with the puppet code. <br />
<br />
With a little more digging we can see which node exactly this failure relates to, e.g we copy the SoftwareDeployment ID from the output above, then run:<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">(undercloud) [stack@undercloud ~]$ openstack software deployment show 421c7860-dd7d-47bd-9e12-de0008a4c106 --format value --column server_id</span>
<span style="color: #888888;">29b3c254-5270-42ae-8150-9fc3f67d3d89</span>
<span style="color: #888888;">(undercloud) [stack@undercloud ~]$ openstack server list | grep 29b3c254-5270-42ae-8150-9fc3f67d3d89</span>
<span style="color: #888888;">| 29b3c254-5270-42ae-8150-9fc3f67d3d89 | overcloud-controller-0 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | oooq_control |</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
Ok so puppet failed while running via ansible on overcloud-controller-0.<br />
<h3>
</h3>
<h3>
Debugging via Ansible directly</h3>
Having identified that the problem was during the ansible-driven configuration phase, one option is to re-run the same configuration directly via ansible-ansible playbook, so you can either increase verbosity or potentially modify the tasks to debug the problem.<br />
<br />
Since the Queens release, this is actually very easy, using a combination of the new "openstack overcloud config download" command and the tripleo dynamic ansible inventory.<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">(undercloud) [stack@undercloud ~]$ openstack overcloud config download</span>
<span style="color: #888888;">The TripleO configuration has been successfully generated into: /home/stack/tripleo-VOVet0-config</span>
<span style="color: #888888;">(undercloud) [stack@undercloud ~]$ cd /home/stack/tripleo-VOVet0-config</span>
<span style="color: #888888;"> (undercloud) [stack@undercloud tripleo-VOVet0-config]$ ls</span>
<span style="color: #888888;"> common_deploy_steps_tasks.yaml external_post_deploy_steps_tasks.yaml </span><span style="color: blue;">templates</span>
<span style="color: blue;"> Compute</span><span style="color: #888888;"> global_vars.yaml update_steps_playbook.yaml</span>
<span style="color: blue;"> Controller group_vars</span><span style="color: #888888;"> update_steps_tasks.yaml</span>
<span style="color: #888888;"> deploy_steps_playbook.yaml post_upgrade_steps_playbook.yaml upgrade_steps_playbook.yaml</span>
<span style="color: #888888;"> external_deploy_steps_tasks.yaml post_upgrade_steps_tasks.yaml upgrade_steps_tasks.yaml</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
Here we can see there is a "deploy_steps_playbook.yaml", which is the entry point to run the ansible service configuration steps. This runs all the common deployment tasks (as outlined above) as well as any service specific tasks (these end up in task include files in the per-role directories, e.g Controller and Compute in this example).<br />
<br />
We can run the playbook again on all nodes with the tripleo-ansible-inventory from tripleo-validations, which is installed by default on the undercloud:<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">(undercloud) [stack@undercloud tripleo-VOVet0-config]$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory deploy_steps_playbook.yaml --limit overcloud-controller-0</span>
<span style="color: #888888;"> ...</span>
<span style="color: #888888;">TASK [Run puppet host configuration for step 1] ********************************************************************</span>
<span style="color: #888888;">ok: [192.168.24.6]</span>
<span style="color: #888888;"></span>
<span style="color: #888888;">TASK [debug] *******************************************************************************************************</span>
<span style="color: #888888;">fatal: [192.168.24.6]: FAILED! => {</span>
<span style="color: #888888;"> "changed": false, </span>
<span style="color: #888888;"> "failed_when_result": true, </span>
<span style="color: #888888;"> "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [</span>
<span style="color: #888888;"> "Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend", </span>
<span style="color: #888888;"> "exception: connect failed", </span>
<span style="color: #888888;"> "Warning: Undefined variable '::deploy_config_name'; ", </span>
<span style="color: #888888;"> " (file & line not available)", </span>
<span style="color: #888888;"> "Warning: Undefined variable 'deploy_config_name'; ", </span>
<span style="color: #888888;"> "Error: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'ugeas' at /etc/puppet/modules/tripleo/manifests/profile
/base/docker.pp:181:5 on node overcloud-controller-0.localdomain"</span>
<span style="color: #888888;"> ]</span>
<span style="color: #888888;">}</span>
<span style="color: #888888;"></span>
<span style="color: #888888;">NO MORE HOSTS LEFT *************************************************************************************************</span>
<span style="color: #888888;"> to retry, use: --limit @/home/stack/tripleo-VOVet0-config/deploy_steps_playbook.retry</span>
<span style="color: #888888;"></span>
<span style="color: #888888;">PLAY RECAP *********************************************************************************************************</span>
<span style="color: #888888;">192.168.24.6 : ok=56 changed=2 unreachable=0 failed=1 </span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
Here we can see the same error is reproduced directly via ansible, and we made use of the --limit option to only run tasks on the overcloud-controller-0 node. We could also have added --tags to limit the tasks further (see <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L472">tripleo-heat-templates</a> for which tags are supported).<br />
<br />
If the error were ansible related, this would be a good way to debug and test any potential fixes to the ansible tasks, and in the upcoming Rocky release there are <a href="http://lists.openstack.org/pipermail/openstack-dev/2018-February/127015.html">plans to switch to this model of deployment by default</a>.<br />
<h3>
</h3>
<h3>
Debugging via Puppet directly</h3>
Since this error seems to be puppet related, the next step is to reproduce it on the host (obviously the steps above often yield enough information to identify the puppet error, but this assumes you need to do more detailed debugging directly via puppet):<br />
<br />
Firstly we log on to the node, and look at the files in the /var/lib/tripleo-config directory.<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">(undercloud) [stack@undercloud tripleo-VOVet0-config]$ ssh heat-admin@192.168.24.6</span>
<span style="color: #888888;">Warning: Permanently added '192.168.24.6' (ECDSA) to the list of known hosts.</span>
<span style="color: #888888;">Last login: Fri Feb 9 14:30:02 2018 from gateway</span>
<span style="color: #888888;">[heat-admin@overcloud-controller-0 ~]$ cd /var/lib/tripleo-config/</span>
<span style="color: #888888;">[heat-admin@overcloud-controller-0 tripleo-config]$ ls</span>
<span style="color: #888888;">docker-container-startup-config-step_1.json docker-container-startup-config-step_4.json puppet_step_config.pp</span>
<span style="color: #888888;">docker-container-startup-config-step_2.json docker-container-startup-config-step_5.json</span>
<span style="color: #888888;">docker-container-startup-config-step_3.json docker-container-startup-config-step_6.json</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
The <b>puppet_step_config.pp</b> file is the manifest applied by ansible on the baremetal host<br />
<br />
We can debug any puppet host configuration by running puppet apply manually.
Note that hiera is used to control the step value, this will be at the same value as the failing step, but it can also be useful sometimes to manually modify this
for development testing of different steps for a particular service.<br />
<br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[root@overcloud-controller-0 tripleo-config]# hiera -c /etc/puppet/hiera.yaml step</span>
<span style="color: #888888;">1</span>
<span style="color: #888888;">[root@overcloud-controller-0 tripleo-config]# cat /etc/puppet/hieradata/config_step.json </span>
<span style="color: #888888;">{"step": 1}[root@overcloud-controller-0 tripleo-config]# puppet apply --debug puppet_step_config.pp</span>
<span style="color: #888888;">...</span>
<span style="color: #888888;">Error: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'ugeas' at /etc/puppet/modules/tripleo/manifests/profile/base/docker.pp:181:5 on node overcloud-controller-0.localdomain</span>
</pre>
<pre style="line-height: 125%; margin: 0;"> </pre>
</div>
<br />
Here we can see the problem is a typo in the /etc/puppet/modules/tripleo/manifests/profile/base/docker.pp file at line 181, I look at the file, fix the problem (ugeas should be augeas) then re-run puppet apply to confirm the fix.<br />
<br />
Note that with puppet module fixes you will need to get the fix either into an updated overcloud image, or <a href="https://hardysteven.blogspot.co.uk/2016/08/tripleo-deploy-artifacts-and-puppet.html">update the module via deploy artifacts</a> for testing local forks of the modules.<br />
<br />
That's all for today, but in a future post, I will cover the new container architecture, and share some debugging approaches I have found helpful when deployment failures are container related. <br />
<br />Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com17tag:blogger.com,1999:blog-7962934533489004582.post-4332316872449864382017-09-27T04:17:00.002-07:002017-09-27T04:18:16.865-07:00OpenStack Days UK<h2>
OpenStack Days UK</h2>
Yesterday I attended the <a href="https://openstackdays.uk/2017/">OpenStack Days UK</a> event, held in London. It was a very good day and there were a number of interesting talks, and it provided a great opportunity to chat with folks about OpenStack.<br />
<br />
I gave a talk, titled "<a href="https://openstackdays.uk/2017/?schedule=deploying-at-scale-tripleo-ansible-containers">Deploying OpenStack at scale, with TripleO, Ansible and Containers</a>", where I gave an update of the recent rework in the <a href="https://docs.openstack.org/developer/tripleo-docs/">TripleO</a> project to make more use of <a href="https://www.ansible.com/">Ansible</a> and enable <a href="https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/index.html">containerized deployments</a>.<br />
<br />
I'm planning some future blog posts with more detail on this topic, but for now here's a copy of the slide deck I used, also available on <a href="https://github.com/hardys/presentations">github</a>.<br />
<br />
<br />
<br />
<iframe allowfullscreen="true" frameborder="0" height="569" mozallowfullscreen="true" src="https://docs.google.com/a/redhat.com/presentation/d/e/2PACX-1vQEcK-_l5PvQtIMVBRj1_2TU3M52F44esHEVSVfDvYVcvfsDFk9JsY6mvGgPgPp5nfCDw2RwMg8s231/embed?start=false&loop=false&delayms=3000" webkitallowfullscreen="true" width="960"></iframe>Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com0tag:blogger.com,1999:blog-7962934533489004582.post-9196725336634175652017-05-11T05:28:00.000-07:002017-05-11T05:28:06.665-07:00OpenStack Summit - TripleO Project OnboardingWe've been having a productive week here in Boston at the <a href="https://www.openstack.org/summit/boston-2017/">OpenStack Summit</a>, and one of the sessions I was involved in was a <a href="https://www.openstack.org/summit/boston-2017/summit-schedule/events/18697/tripleo-project-onboarding">TripleO project Onboarding session</a>.<br />
<br />
The project onboarding sessions are a new idea for this summit, and provide the opportunity for new or potential contributors (and/or users/operators) to talk with the existing project developers and get tips on how to get started as well as ask any questions and discuss ideas/issues.<br />
<br />
The TripleO session went well, and I'm very happy to report it was well attended and we had some good discussions. The session was informal with an emphasis on questions and some live demos/examples, but we did also use a few slides which provide an overview and some context for those new to the project.<br />
<br />
Here are the slides used (also on <a href="https://github.com/hardys/presentations">my github</a>), unfortunately I can't share the Q+A aspects of the session as it wasn't recorded, but I hope the slides will prove useful - we can be found in #tripleo on Freenode if anyone has questions about the slides or getting started with TripleO in general.<br />
<br />
<iframe allowfullscreen="true" frameborder="0" height="569" mozallowfullscreen="true" src="https://docs.google.com/presentation/d/15ldFtsphNwetrkZ5npKyOpw2ooJ5X6W6F_xQFHFmHpY/embed?start=false&loop=false&delayms=3000" webkitallowfullscreen="true" width="960"></iframe>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com7tag:blogger.com,1999:blog-7962934533489004582.post-61038390305757396892017-03-03T02:51:00.000-08:002017-03-03T07:59:06.535-08:00Developing Mistral workflows for TripleODuring the newton/ocata development cycles, <a href="https://docs.openstack.org/developer/tripleo-docs/">TripleO</a> made changes to the architecture so we make use of <a href="https://docs.openstack.org/developer/mistral/">Mistral</a> (the OpenStack workflow API project) to drive workflows required to deploy your OpenStack cloud.<br />
<br />
Prior to this change we had workflow defined inside <a href="https://github.com/openstack/python-tripleoclient">python-tripleoclient</a>, and most API calls were made directly to <a href="https://docs.openstack.org/developer/heat/">Heat</a>. This worked OK but there was too much "business logic" inside the client, which doesn't work well if non-python clients (such as tripleo-ui) want to interact with <a href="https://docs.openstack.org/developer/tripleo-docs/">TripleO</a>.<br />
<br />
To solve this problem, number of mistral workflows and custom actions have been implemented, which are available via the Mistral API on the undercloud. This can be considered the primary "TripleO API" for driving all deployment tasks now.<br />
<br />
Here's a diagram showing how it fits together:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAiZmDnP2QnPzBXEAdMqmCPXMaRWyBxgQf7pv2V3su0QG_dVMJPP6yuq65OvHql0dnqTvzvqNvIw-nhwADplMHduZmc4aVV8jf-LuyJUiPToZ55NXeKa0dKNsOefDP7ez1lj5RZnFKeLI/s1600/Mistral+API.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="398" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAiZmDnP2QnPzBXEAdMqmCPXMaRWyBxgQf7pv2V3su0QG_dVMJPP6yuq65OvHql0dnqTvzvqNvIw-nhwADplMHduZmc4aVV8jf-LuyJUiPToZ55NXeKa0dKNsOefDP7ez1lj5RZnFKeLI/s640/Mistral+API.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Overview of Mistral integration in TripleO</td></tr>
</tbody></table>
<br />
<br />
<h3>
Mistral workflows and actions</h3>
There are two primary interfaces to mistral, <a href="https://docs.openstack.org/developer/mistral/dsl/dsl_v2.html#workflows"><b><i>workflows</i></b></a> which are a yaml definition of a process or series of tasks, and <a href="https://docs.openstack.org/developer/mistral/dsl/dsl_v2.html#actions"><i><b>actions</b></i></a> which are a concrete definition of how to do a specific task (such as call some OpenStack API).<br />
<br />
Workflows and actions can defined directly via the <a href="https://docs.openstack.org/developer/mistral/developer/webapi/index.html">mistral API</a>, or a wrapper called a <a href="https://docs.openstack.org/developer/mistral/dsl/dsl_v2.html#workbooks"><b><i>workbook</i></b></a>. Mistral actions are also defined via a <a href="https://docs.openstack.org/developer/mistral/developer/creating_custom_action.html">python plugin interface</a>, which TripleO uses to run some tasks such as running <a href="http://jinja.pocoo.org/">jinja2</a> on <a href="https://github.com/openstack/tripleo-heat-templates">tripleo-heat-templates</a> prior to calling <a href="https://docs.openstack.org/developer/heat/">Heat</a> to orchestrate the deployment.<br />
<br />
<h3>
Mistral workflows, in detail</h3>
Here I'm going to show how to view and interact with the mistral workflows used by TripleO directly, which is useful to understand what TripleO is doing "under the hood" during a deployment, and also for debugging/development.<br />
<br />
First we view the mistral workbooks loaded into Mistral - these contain the TripleO specific workflows and are defined in <a href="https://github.com/openstack/tripleo-common/tree/master/workbooks">tripleo-common</a><br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[stack@undercloud ~]$ . stackrc </span>
<span style="color: #888888;">[stack@undercloud ~]$ mistral workbook-list</span>
<span style="color: #888888;">+----------------------------+--------+---------------------+------------+</span>
<span style="color: #888888;">| Name | Tags | Created at | Updated at |</span>
<span style="color: #888888;">+----------------------------+--------+---------------------+------------+</span>
<span style="color: #888888;">| tripleo.deployment.v1 | <none> | 2017-02-27 17:59:04 | None |</span>
<span style="color: #888888;">| tripleo.package_update.v1 | <none> | 2017-02-27 17:59:06 | None |</span>
<span style="color: #888888;">| tripleo.plan_management.v1 | <none> | 2017-02-27 17:59:09 | None |</span>
<span style="color: #888888;">| tripleo.scale.v1 | <none> | 2017-02-27 17:59:11 | None |</span>
<span style="color: #888888;">| tripleo.stack.v1 | <none> | 2017-02-27 17:59:13 | None |</span>
<span style="color: #888888;">| tripleo.validations.v1 | <none> | 2017-02-27 17:59:15 | None |</span>
<span style="color: #888888;">| tripleo.baremetal.v1 | <none> | 2017-02-28 19:26:33 | None |</span>
<span style="color: #888888;">+----------------------------+--------+---------------------+------------+</span>
</pre>
</div>
<br />
The name of the workbook constitutes a namespace for the workflows it contains, so we can view the related workflows using grep (I also grep for tag_node to reduce the number of matches).<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[stack@undercloud ~]$ mistral workflow-list | grep "tripleo.baremetal.v1" | grep tag_node</span>
<span style="color: #888888;">| 75d2566c-13d9-4aa3-b18d-8e8fc0dd2119 | tripleo.baremetal.v1.tag_nodes | 660c5ec71ce043c1a43d3529e7065a9d | <none> | tag_node_uuids, untag_nod... | 2017-02-28 19:26:33 | None |</span>
<span style="color: #888888;">| 7a4220cc-f323-44a4-bb0b-5824377af249 | tripleo.baremetal.v1.tag_node | 660c5ec71ce043c1a43d3529e7065a9d | <none> | node_uuid, role=None, que... | 2017-02-28 19:26:33 | None |</span> </pre>
</div>
<br />
When you know the name of a workflow, you can inspect the required inputs, and run it directly via a mistral execution, in this case we're running the tripleo.baremetal.v1.tag_node workflow, which modifies the profile assigned in the ironic node capabilities (see tripleo-docs for more information about <a href="https://docs.openstack.org/developer/tripleo-docs/advanced_deployment/profile_matching.html#manual-profile-tagging">manual tagging of nodes</a>)<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[stack@undercloud ~]$ mistral workflow-get tripleo.baremetal.v1.tag_node</span>
<span style="color: #888888;">+------------+------------------------------------------+</span>
<span style="color: #888888;">| Field | Value |</span>
<span style="color: #888888;">+------------+------------------------------------------+</span>
<span style="color: #888888;">| ID | 7a4220cc-f323-44a4-bb0b-5824377af249 |</span>
<span style="color: #888888;">| Name | tripleo.baremetal.v1.tag_node |</span>
<span style="color: #888888;">| Project ID | 660c5ec71ce043c1a43d3529e7065a9d |</span>
<span style="color: #888888;">| Tags | <none> |</span>
<span style="color: #888888;">| Input | node_uuid, role=None, queue_name=tripleo |</span>
<span style="color: #888888;">| Created at | 2017-02-28 19:26:33 |</span>
<span style="color: #888888;">| Updated at | None |</span>
<span style="color: #888888;">+------------+------------------------------------------+</span>
<span style="color: #888888;">[stack@undercloud ~]$ ironic node-list</span>
<span style="color: #888888;">+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+</span>
<span style="color: #888888;">| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |</span>
<span style="color: #888888;">+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+</span>
<span style="color: #888888;">| 30182cb9-eba9-4335-b6b4-d74fe2581102 | control-0 | None | power off | available | False |</span>
<span style="color: #888888;">| 19fd7ea7-b4a0-4ae9-a06a-2f3d44f739e9 | compute-0 | None | power off | available | False |</span>
<span style="color: #888888;">+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+</span>
<span style="color: #888888;">[stack@undercloud ~]$ mistral execution-create tripleo.baremetal.v1.tag_node '{"node_uuid": "30182cb9-eba9-4335-b6b4-d74fe2581102", "role": "test"}'</span>
<span style="color: #888888;">+-------------------+--------------------------------------+</span>
<span style="color: #888888;">| Field | Value |</span>
<span style="color: #888888;">+-------------------+--------------------------------------+</span>
<span style="color: #888888;">| ID | 6a141065-ad6e-4477-b1a8-c178e6fcadcb |</span>
<span style="color: #888888;">| Workflow ID | 7a4220cc-f323-44a4-bb0b-5824377af249 |</span>
<span style="color: #888888;">| Workflow name | tripleo.baremetal.v1.tag_node |</span>
<span style="color: #888888;">| Description | |</span>
<span style="color: #888888;">| Task Execution ID | <none> |</span>
<span style="color: #888888;">| State | RUNNING |</span>
<span style="color: #888888;">| State info | None |</span>
<span style="color: #888888;">| Created at | 2017-03-03 09:53:10 |</span>
<span style="color: #888888;">| Updated at | 2017-03-03 09:53:10 |</span>
<span style="color: #888888;">+-------------------+--------------------------------------+</span>
</pre>
</div>
<br />
At this point the mistral workflow is running, and it'll either succeed or fail, and also create some output (which in the TripleO model is sometimes returned to the Ux via a <a href="https://docs.openstack.org/developer/zaqar/">Zaqar</a> queue). We can view the status, and the outputs (truncated for brevity):<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[stack@undercloud ~]$ mistral execution-list | grep 6a141065-ad6e-4477-b1a8-c178e6fcadcb</span>
<span style="color: #888888;">| 6a141065-ad6e-4477-b1a8-c178e6fcadcb | 7a4220cc-f323-44a4-bb0b-5824377af249 | tripleo.baremetal.v1.tag_node | | <none> | SUCCESS | None | 2017-03-03 09:53:10 | 2017-03-03 09:53:11 |</span>
<span style="color: #888888;">[stack@undercloud ~]$ mistral execution-get-output 6a141065-ad6e-4477-b1a8-c178e6fcadcb</span>
<span style="color: #888888;">{</span>
<span style="color: #888888;"> "status": "SUCCESS", </span>
<span style="color: #888888;"> "message": {</span>
<span style="color: #888888;">...</span>
</pre>
</div>
<br />
So that's it - we ran a mistral workflow, it suceeded and we looked at the output, now we can see the result looking at the node in Ironic, it worked! :) <br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[stack@undercloud ~]$ ironic node-show 30182cb9-eba9-4335-b6b4-d74fe2581102 | grep profile</span>
<span style="color: #888888;">| | u'cpus': u'2', u'capabilities': u'<b>profile:test</b>,cpu_hugepages:true,boot_o |</span>
</pre>
</div>
<h3>
</h3>
<h3>
Mistral workflows, create your own!</h3>
Here I'll show how to develop your own custom workflows (which isn't something we expect operators to necessarily do, but is now part of many developers
workflow during feature development for TripleO).<br />
<br />
First, we create a simple yaml definition of the workflow, as defined in the <a href="https://docs.openstack.org/developer/mistral/dsl/dsl_v2.html#workflows">v2 Mistral DSL</a> - <a href="https://gist.github.com/hardys/b1999c500185ccc832f0ac8731174b2a">this example</a> lists all available ironic nodes, then finds those which match the "test" profile we assigned in the example above:<br />
<br />
<script src="https://gist.github.com/hardys/b1999c500185ccc832f0ac8731174b2a.js"></script>
<!-- HTML generated using hilite.me --><br />
<!-- HTML generated using hilite.me -->This example uses the mistral built-in "ironic" action, which is basically a pass-through action exposing the python-ironicclient interfaces. Similar actions exist for the majority of OpenStack python clients, so this is a pretty flexible interface.<br />
<br />
Now we can now upload the workflow (not wrapped in a workbook this time, so we use workflow-create), run it via execution create, then look at the outputs - we can see that the <i>matching_nodes</i> output matches the ID of the node we tagged in the example above - success! :)<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">[stack@undercloud tripleo-common]$ mistral workflow-create shtest.yaml </span>
<span style="color: #888888;">+--------------------------------------+-------------------------+----------------------------------+--------+--------------+---------------------+------------+</span>
<span style="color: #888888;">| ID | Name | Project ID | Tags | Input | Created at | Updated at |</span>
<span style="color: #888888;">+--------------------------------------+-------------------------+----------------------------------+--------+--------------+---------------------+------------+</span>
<span style="color: #888888;">| 2b8f2bea-f3dd-42f0-ad16-79987c75df4d | test_nodes_with_profile | 660c5ec71ce043c1a43d3529e7065a9d | <none> | profile=test | 2017-03-03 10:18:48 | None |</span>
<span style="color: #888888;">+--------------------------------------+-------------------------+----------------------------------+--------+--------------+---------------------+------------+</span>
<span style="color: #888888;">[stack@undercloud tripleo-common]$ mistral execution-create test_nodes_with_profile</span>
<span style="color: #888888;">+-------------------+--------------------------------------+</span>
<span style="color: #888888;">| Field | Value |</span>
<span style="color: #888888;">+-------------------+--------------------------------------+</span>
<span style="color: #888888;">| ID | 2392ed1c-96b4-4787-9d11-0f3069e9a7e5 |</span>
<span style="color: #888888;">| Workflow ID | 2b8f2bea-f3dd-42f0-ad16-79987c75df4d |</span>
<span style="color: #888888;">| Workflow name | test_nodes_with_profile |</span>
<span style="color: #888888;">| Description | |</span>
<span style="color: #888888;">| Task Execution ID | <none> |</span>
<span style="color: #888888;">| State | RUNNING |</span>
<span style="color: #888888;">| State info | None |</span>
<span style="color: #888888;">| Created at | 2017-03-03 10:19:30 |</span>
<span style="color: #888888;">| Updated at | 2017-03-03 10:19:30 |</span>
<span style="color: #888888;">+-------------------+--------------------------------------+</span>
<span style="color: #888888;">[stack@undercloud tripleo-common]$ mistral execution-list | grep 2392ed1c-96b4-4787-9d11-0f3069e9a7e5</span>
<span style="color: #888888;">| 2392ed1c-96b4-4787-9d11-0f3069e9a7e5 | 2b8f2bea-f3dd-42f0-ad16-79987c75df4d | test_nodes_with_profile | | <none> | SUCCESS | None | 2017-03-03 10:19:30 | 2017-03-03 10:19:31 |</span>
<span style="color: #888888;">[stack@undercloud tripleo-common]$ mistral execution-get-output 2392ed1c-96b4-4787-9d11-0f3069e9a7e5</span>
<span style="color: #888888;">{</span>
<span style="color: #888888;"> "matching_nodes": [</span>
<span style="color: #888888;"> "30182cb9-eba9-4335-b6b4-d74fe2581102"</span>
<span style="color: #888888;"> ], </span>
<span style="color: #888888;"> "available_nodes": [</span>
<span style="color: #888888;"> "30182cb9-eba9-4335-b6b4-d74fe2581102", </span>
<span style="color: #888888;"> "19fd7ea7-b4a0-4ae9-a06a-2f3d44f739e9"</span>
<span style="color: #888888;"> ]</span>
<span style="color: #888888;">}</span>
</pre>
</div>
<br />
Using this basic example, you can see how to develop workflows which can then easily be copied into the tripleo-common workbooks, and integrated into the TripleO deployment workflow.<br />
<br />
In a future post, I'll dig into the use of custom actions, and how to develop/debug those.Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com14tag:blogger.com,1999:blog-7962934533489004582.post-43331220304783110132016-10-10T02:04:00.000-07:002016-10-25T02:32:26.703-07:00TripleO composable/custom rolesThis is a follow-up to my <a href="http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html" target="_blank">previous post outlining the new composable services interfaces </a>, which covered the basics of the new for <a href="https://blueprints.launchpad.net/tripleo/newton" target="_blank">Newton</a> composable services model.<br />
<br />
The final piece of the composability model we've been developing this cycle is the ability to deploy <a href="https://blueprints.launchpad.net/tripleo/+spec/custom-roles" target="_blank">user-defined custom roles</a>, in addition to (or even instead of) the built in TripleO roles (where a role is a group of servers, e.g "Controller", which runs some combination of services).<br />
<br />
What follows is an overview of this new functionality, the primary interfaces, and some usage examples and a summary of future planned work.<br />
<br />
<br />
<a name='more'></a><br />
<h2>
Fully Composable/Custom Roles </h2>
<style type="text/css">p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }</style>
As described in <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-1-roles-and.html" target="_blank">previous
posts</a> TripleO has for a long time provided a fixed architecture
with 5 roles (where "roles" means groups of nodes) e.g
Controller, Compute, BlockStorage, CephStorage and ObjectStorage.<br />
<br />
This architecture has been sufficient to enable standardized
deployments, but it's not very flexible. With the
addition of the <a href="http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html">composable-services model</a>, moving services around
between these roles becomes much easier, but many operators want to
go further, and have full control of service placement on any
arbitrary roles.<br />
<br />
Now that the <a href="https://blueprints.launchpad.net/tripleo/+spec/custom-roles" target="_blank">custom-roles
feature</a> has been implemented, this is possible, and operators can
define arbitrary role types to enable fully composable deployments.
When combined with <a href="http://hardysteven.blogspot.co.uk/2016/08/tripleo-composable-services-101.html" target="_blank">composable
services</a> represents a huge step forward for TripleO flexibility!
:)<br />
<br />
<h2>
<style type="text/css">h2.cjk { font-family: "Droid Sans Fallback"; }h2.ctl { font-family: "Lohit Devanagari"; }p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }</style>
</h2>
<h2 class="western">
Usage examples</h2>
To deploy with additional custom roles (or to remove/rename the
default roles), a new interface has been added to the
python-tripleoclient “overcloud deploy interface”, so you simply
need to copy the <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/roles_data.yaml">default roles_data.yaml,</a> modify to suit your
requirements (for example by moving services between roles, or adding
a new role), then do a deployment referencing the modified
<a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/roles_data.yaml">roles_data.yaml</a> file:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
my_roles_data.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"><modify my_roles_data.yaml></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;">openstack overcloud deploy –templates -r my_roles_data.yaml</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="line-height: 100%; margin-bottom: 0cm;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0cm;">
Alternatively you
can copy the entire tripleo-heat-templates tree (or use a git
checkout):</div>
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">cp -r /usr/share/openstack-tripleo-heat-templates my-tripleo-heat-templates </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><modify my-tripleo-heat-templates/roles_data.yaml><br />openstack overcloud deploy –templates my-tripleo-heat-templates </span><br />
<br />
Both approaches are essentially equivalent, the -r option simply
overwrites the default roles_data.yaml during creation of the plan
data (stored in swift on the undercloud), but it's slightly more
convenient if you want to use the default packaged
tripleo-heat-templates instead of constantly rebasing a copied tree. <br />
<br />
So, lets say you wanted to deploy one additional node, only running the OS::TripleO::Ntp composable service, you'd copy roles_data.yaml, and append a list entry like this:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">- name: NtpRole<br /> CountDefault: 1<br /> ServicesDefault:<br /> - OS::TripleO::Services::Ntp</span><br />
<br />
<br />
<i>(Note that in practice you'll probably also want some of the common services deployed on all roles, such as OS::TripleO::Services::Kernel, OS::TripleO::Services::TripleoPackages, OS::TripleO::Services::TripleoFirewall and OS::TripleO::Services::VipHosts)</i><br />
<h2>
</h2>
<h2>
Nice, so how does it work?</h2>
<style type="text/css">p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }</style>
<br />
The main change made to enable custom roles is a pre-deployment
templating step which runs<a href="http://jinja.pocoo.org/docs/"> Jinja2</a>. We define a <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/roles_data.yaml">roles_data.yaml file</a>(which can be overridden by the user), which contains a list of role
names, and optionally some additional data related to default
parameter values (such as the default services deployed on the role,
and default count in the group)<br />
<br />
The <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/roles_data.yaml">roles_data.yaml definitions</a> look like this:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">- name: Controller</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> CountDefault: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> ServicesDefault:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> - OS::TripleO::Services::CACerts</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> - OS::TripleO::Services::CephMon</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span> <span style="font-family: "courier new" , "courier" , monospace;">- OS::TripleO::Services::CinderApi</span><span style="font-family: "courier new" , "courier" , monospace;"></span><br />
- ... <br />
<br />
The format is simply a yaml list of maps, with a mandatory “name”
key in each map, and a number of optional FooDefault keys which set
the parameter defaults for the role (as a convenience so the user
won't have to specify it via an environment file during the overcloud
deployment).
<br />
<br />
A <a href="http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/actions">custom mistral action</a> is used to run Jinja2 when creating or
updating a “deployment plan” (which is a combination of some heat
templates stored in swift, and a mistral environment containing user
parameters) – and this basically consumes the <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/roles_data.yaml">roles_data.yaml list</a> of required roles, and outputs a rendered tree of Heat templates
ready to deploy your overcloud.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvtJayR-sseLE1DvuRPeK7oStFmH8v8IYuLb5BuEel93hCTvPbfV_8VB0yKWKyGv50PF7rq2DOV6L_GPocDZQU-pOQvAxZZZXo_6fnWWEmMR_C5z9xx0x-EE1-xhtocYU_lMP-lukrs6w/s1600/Custom+Roles+Overview%25281%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="444" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvtJayR-sseLE1DvuRPeK7oStFmH8v8IYuLb5BuEel93hCTvPbfV_8VB0yKWKyGv50PF7rq2DOV6L_GPocDZQU-pOQvAxZZZXo_6fnWWEmMR_C5z9xx0x-EE1-xhtocYU_lMP-lukrs6w/s640/Custom+Roles+Overview%25281%2529.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Custom Roles, overview</td></tr>
</tbody></table>
<br />
<style type="text/css">p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }</style>
<br />
There are two types of Jinja2 templates which are rendered differently, distinguished by
the file extension/suffix:<br />
<br />
<h3>
<b>foo.<span style="color: red;">j2.yaml</span></b></h3>
<h3>
</h3>
This will pass in the contents of the <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/roles_data.yaml">roles_data.yaml list</a>, and
iterate over each role in the list, The resulting file in the plan
swift container will be named foo.yaml.<br />
Here's an example of the syntax used for j2 templating inside
these files:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">enabled_services:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> list_join:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> - ','</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;">{% for role in roles %}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;"> - {get_attr: [{{role.name}}ServiceChain, role_data,
service_names]}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;">{% endfor %}</span><br />
<br />
This example is from <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud.j2.yaml">overcloud.j2.yaml</a>, it does a jinja2 loop
appending service_names for all roles *ServiceChain resources (which
are also dynamically generated via a similar loop), which is then
processed on deployment via a heat list_join function,<br />
<br />
<h3>
<b>foo.<span style="color: red;">role.j2.yaml</span></b></h3>
This will generate a file per-role, where only the name of the
role is passed in during the templating step, with the resulting
files being called rolename-foo.yaml. <i>(Note that If you have a
role which requires a special template, it is possible to disable
th</i><i>is</i><i> file generation by adding the </i><i>path</i><i>
to the<a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/j2_excludes.yaml"> j2_excludes.yaml file</a>)</i><br />
<div style="font-style: normal;">
<br /></div>
<div style="font-style: normal;">
Here's an example of the syntax used in
these files (taken from the <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/role.role.j2.yaml">role.role.j2.yaml file</a>, which is our new
definition of server for a generic role):</div>
<div style="font-style: normal;">
<br /></div>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;">resources:</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> {{role}}:</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> type: OS::TripleO::Server</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> metadata:</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> os-collect-config:</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> command: {get_param:
ConfigCommand}</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> properties:</span></div>
<span style="font-family: "courier new" , "courier" , monospace;">
</span>
<div style="font-style: normal;">
<span style="font-family: "courier new" , "courier" , monospace;"> image: {get_param: {{role}}Image}</span></div>
<div style="font-style: normal;">
<br /></div>
<div style="font-style: normal;">
As you can see, this simply allows use
of a {{role}} placeholder, which is then substituted with the role
name when rendering each file (one file per role defined in the
roles_data.yaml list).</div>
<br />
<style type="text/css">h2.cjk { font-family: "Droid Sans Fallback"; }h2.ctl { font-family: "Lohit Devanagari"; }p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }</style>
<br />
<h2 class="western">
Debugging/Development tips</h2>
When making changes to either the roles_data.yaml, and
particularly when making changes to the *.j2.yaml files in
tripleo-heat-templates, it's often helpful to view the rendered
templates before any overcloud deployment is attempted.<br />
<br />
This is possible via use of the “openstack overcloud plan
create” interface (which doesn't yet support the -r option above,
so you have to copy or git clone the tree), combined with
swiftclient:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">openstack overcloud plan create overcloud –templates
my_tripleo_heat_templates</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;">mkdir tmp_templates && pushd tmp_templates</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">
</span><span style="font-family: "courier new" , "courier" , monospace;">swift download overcloud</span><br />
<br />
This will download the full tree of rendered files from the swift
container (named “overcloud” due to the name passed to plan
create), so you can e.g view the rendered overcloud.yaml that's
generated by combining the <a href="http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud.j2.yaml">overcloud.j2.yaml</a> template with the
roles_data.yaml file.<br />
<br />
If you make a mistake in your *.j2.yaml file, the jinja2 error
should be returned via the plan create command, but it can also be
useful to tail -f /var/log/mistral/mistral-server.log for additional
information during development (this shows the output logged from
running jinja2 via the custom mistral action plugin).<br />
<br />
<h2 class="western">
Limitations/future work</h2>
These new interfaces allow for much greater deployment flexibility
and choice, but there are a few remaining issues which will be
addressed in future development cycles:<br />
<ol>
<li>
All services managed by pacemaker are still tied to the
Controller role. Thanks to the implementation of a more <a href="https://blueprints.launchpad.net/tripleo/+spec/ha-lightweight-architecture" target="_blank">lightweight HA architecture </a>during the Newton
cycle, the list of services managed by pacemaker is considerably
reduced, but there's still a number of services (DB & RPC
services primarily) which are, and until the <a href="https://blueprints.launchpad.net/tripleo/+spec/composable-ha" target="_blank">composable-ha blueprint</a>
is completed (hopefully during Ocata), these services cannot be
moved to a non Controller role. </li>
<li>Custom isolated networks cannot be defined. Since
arbitrary roles types can now be defined, there may be a requirement
to define arbitrary additional networks for network-isolation, but
right now this is not possible.</li>
<li>roles_data.yaml must be copied. As in the examples above,
it's necessary to copy either roles_data.yaml, (or the entire
tripleo-heat-templates tree), which means if the packaged
roles_data.yaml changes (such as to add new services to the built-in
roles), you must merge these changes in with your custom roles_data.
In future we may add a convenience interface which makes it easier
to e.g add a new role without having to care about the default role
definitions.</li>
<li>No model for dependencies between services. Currently ensuring the right combination of services is deployed on specific roles is left to the operator, there's no validation of incompatible or inter-dependent services, but this may be addressed in a future release.<br />
</li>
</ol>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com9tag:blogger.com,1999:blog-7962934533489004582.post-62219628721830617682016-09-01T02:31:00.000-07:002016-09-01T04:34:04.531-07:00Complex data transformations with nested Heat intrinsic functionsDisclaimer, what follows is either pretty neat, or pure-evil depending your your viewpoint ;) But it's based on a <a href="https://review.openstack.org/#/c/361778/" target="_blank">real use-case </a>and it works, so I'm posting this to document the approach, why it's needed, and hopefully stimulate some discussion around optimizations leading to a improved/simplified implementation in the future.<br />
<br />
<br />
<a name='more'></a><br />
<br />
<h2>
The requirement</h2>
In TripleO we have a requirement enable composition of different services onto different roles (groups of physical nodes), we need input data to configure the services which combines knowledge of the enabled services, which nodes/role they're running on, and which overlay network each service is bound to.<br />
<br />
To do this, we need to input several pieces of data:<br />
<br />
1. A list of the OpenStack services enabled for a particular deployment, expressed as a heat parameter it looks something like this:<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> EnabledServices:<br /> type: comma_delimited_list<br /> default:<br /> - heat_api</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - heat_engine<br /> - nova_api<br /> - neutron_api<br /> - glance_api<br /> - ceph_mon</span><br />
<br />
2. A mapping of service names to one of several isolated overlay networks, such as "internal_api" "external" or "storage" etc:<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> ServiceNetMap:<br /> type: json<br /> default:<br /> heat_api_network: internal_api<br /> nova_api_network: internal_api<br /> neutron_api_network: internal_api<br /> glance_api_network: storage<br /> ceph_mon_network: storage</span><br />
<br />
3. A mapping of the network names to the actual IP address (either a single VIP pointing to a loadbalancer, or a list of the IPs bound to that network for all nodes running the service):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> NetIpMap:<br /> type: json<br /> default:<br /> internal_api: 192.168.1.12<br /> storage: 192.168.1.13</span><br />
<br />
<h2>
The implementation, step by step</h2>
<br />
<br />
<h3>
Dynamically generate an initial mapping for all enabled services</h3>
Here we can use a nice pattern which combines the heat <a href="http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#repeat" target="_blank">repeat</a> function with <a href="http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#map-merge" target="_blank">map_merge</a>:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> map_merge:<br /> repeat:<br /> template:<br /> SERVICE_ip: SERVICE_network<br /> for_each:<br /> SERVICE: {get_param: EnabledServices}</span><br />
<br />
<br />
<br />
<b>Step1</b>: repeat dynamically generates lists (including lists of maps as in this case), so we use it to generate a list of maps for every service in the EnabledServices list with a placeholder for the network, e.g:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> - heat_api_ip: heat_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <span style="font-family: "courier new" , "courier" , monospace;"> - heat_engine_ip: heat_engine_network</span> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - nova_api_ip: nova_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - neutron_api_ip: neutron_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - glance_api_ip: glance_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - ceph_mon_ip: ceph_mon_network</span><br />
<br />
<b>Step2</b>: map_merge combines this list of maps with only one key to one big map for all EnabledServices<br />
<br />
<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> heat_api_ip: heat_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> heat_engine_ip: heat_engine_network</span> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nova_api_ip: nova_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> neutron_api_ip: neutron_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> glance_api_ip: glance_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ceph_mon_ip: ceph_mon_network</span><br />
<br />
<h3>
Substitute placeholder for the actual network/IP</h3>
<br />
We approach this in two passes, with two nested <a href="http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#map-replace" target="_blank">map_replace</a> calls (a new function I wrote for newton Heat which can do key/value substitutions on any mapping):<br />
<br />
<br />
<br />
map_replace:<br />
- map_replace:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> - heat_api_ip: heat_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> heat_engine_ip: heat_engine_network</span> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nova_api_ip: nova_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> neutron_api_ip: neutron_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> glance_api_ip: glance_api_network</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ceph_mon_ip: ceph_mon_network</span><br />
- values: {get_param: ServiceNetMap}<br />
- values: {get_param: NetIpMap}<br />
<br />
<br />
<b>Step3</b>: The inner map_replace substitutes the placeholder into the actual network provided in the ServiceNetMap mapping, which gives e.g<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> heat_api_ip: internal_api</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> heat_engine_ip: heat_engine_network</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nova_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">internal_api</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> neutron_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">internal_api</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> glance_api_ip: storage</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ceph_mon_ip: storage</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><span style="font-family: inherit;"> </span><br />
<span style="font-family: inherit;">Note that if there's no network assigned in ServiceNetMap for the service, no replacement will occur, so the value will remain e.g heat_engine_network, more on this later..</span><br />
<br />
<b>Step4</b>: the outer map_replace substitutes the network name, e.g internal_api, with the actual VIP for that network provided by the ServiceNetMap mapping, which gives the final mapping of:<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> heat_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"> heat_engine_ip: </span></span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;">heat_engine_network</span></span> </span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nova_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> neutron_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> glance_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.13</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ceph_mon_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.13</span><br />
<br />
<h3>
<span style="font-family: "courier new" , "courier" , monospace;"></span>Filter any values we don't want</h3>
As you can see we got a value we don't want - heat_engine is like many non-api services in that it's not bound to any network, it only talks to rabbitmq, so we don't have any entry in ServiceNetMap for it.<br />
<br />
We can therefore remove any entries which remain in the mapping using the <a href="http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#yaql" target="_blank">yaql</a> heat function, which is an interface to run <a href="https://yaql.readthedocs.io/en/latest/getting_started.html#basic-yaql-query-operations" target="_blank">yaql queries</a> inside a heat template. <br />
<br />
It has to be said yaql is very powerful, but the docs are pretty sparse (but improving), so I tend to read the <a href="https://github.com/openstack/yaql/tree/master/yaql/tests" target="_blank">unit tests</a> instead of the docs for usage examples.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> yaql:<br /> expression: dict($.data.map.items().where(isString($[1]) and not $[1].endsWith("_network")))<br /> data:<br /> map:</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> heat_api_ip: 192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> heat_engine_ip: heat_engine_network </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nova_api_ip: 192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> neutron_api_ip: 192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> glance_api_ip: 192.168.1.13</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ceph_mon_ip: 192.168.1.13</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<br />
<b>Step5</b>: filter all map values where the value is a string, and the string ends with "_network" via yaql, which gives:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> heat_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span><span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "courier new" , "courier" , monospace;"></span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nova_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> neutron_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.12</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> glance_api_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.13</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ceph_mon_ip: </span><span style="font-family: "courier new" , "courier" , monospace;">192.168.1.13</span><br />
<br />
<br />
So, that's it - we now transformed two input maps and a list into a dynamically generated mapping based on the list items! :)<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<h2>
</h2>
<h2>
Implementation, completed</h2>
<br />
Pulling all of the above together, here's a full example (you'll need a newton Heat environment to run this), it combines all steps described above into one big combination of nested intrinsic functions:<br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<b>Edit </b>- also <a href="https://github.com/hardys/demo_templates/blob/master/intrinsic_functions_blog/nested_functions.yaml">available on github</a><span style="font-family: "courier new" , "courier" , monospace;"><br /></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">heat_template_version: 2016-10-14<br /><br />description: ><br /> Example of nested heat functions<br /><br />parameters:<br /> NetIpMap:<br /> type: json<br /> default:<br /> internal_api: 192.168.1.12<br /> storage: 192.168.1.13<br /><br /> EnabledServices:<br /> type: comma_delimited_list<br /> default:<br /> - heat_api<br /> - nova_api<br /> - neutron_api<br /> - glance_api<br /> - ceph_mon<br /><br /> ServiceNetMap:<br /> type: json<br /> default:<br /> heat_api_network: internal_api<br /> nova_api_network: internal_api<br /> neutron_api_network: internal_api<br /> glance_api_network: storage<br /> ceph_mon_network: storage<br /><br /><br />outputs:<br /> service_ip_map:<br /> description: Mapping of service names to IP address for the assigned network<br /> value:<br /> yaql:<br /> expression: dict($.data.map.items().where(isString($[1]) and not $[1].endsWith("_network")))<br /> data:<br /> map:<br /> map_replace:<br /> - map_replace:<br /> - map_merge:<br /> repeat:<br /> template:<br /> SERVICE_ip: SERVICE_network<br /> for_each:<br /> SERVICE: {get_param: EnabledServices}<br /> - values: {get_param: ServiceNetMap}<br /> - values: {get_param: NetIpMap}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<h3>
</h3>
<h3>
<span style="font-family: "courier new" , "courier" , monospace;"> </span></h3>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com12tag:blogger.com,1999:blog-7962934533489004582.post-15255949371196460822016-08-12T15:20:00.001-07:002016-08-12T15:20:35.621-07:00TripleO Deploy Artifacts (and puppet development workflow)<br /><br />For a while now, TripleO has supported a "DeployArtifacts" interface, aimed at making it easier to deploy modified/additional files on your overcloud, without the overhead of frequently rebuilding images.<br /><br />This started out as a way to enable faster iteration on puppet module development (the puppet modules are by default stored inside the images deployed by TripleO, and generally you'll want to do development in a git checkout on the undercloud node), but it is actually a generic interface that can be used for a variety of deployment time customizations.<br /><br /><br /><a name='more'></a><br /><br /><h2>
Ok, how do I use it?</h2>
<br />Lets start with a couple of usage examples, making use of some helper scripts that are maintained in the tripleo-common repo (in future similar helper interfaces may be added to the TripleO CLI/UI but right now this is more targetted at developers and advanced operator usage).<br /><br />First clone the <b>tripleo-common</b> repo (you can skip this step if you're running a packaged version which already contains the following scripts):<br />
<br />
<br />
<br />
<br />
<br /><span style="font-family: "Courier New",Courier,monospace;">[stack@instack ~]$ git clone <a href="https://git.openstack.org/openstack/tripleo-common">https://git.openstack.org/openstack/tripleo-common</a></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<br />
<br />
<br />There are two scripts of interest, firstly a generic script that can be used to deploy any kind of file (aka artifact) <a href="https://github.com/openstack/tripleo-common/blob/master/scripts/upload-puppet-modules" target="_blank"><b><i>tripleo-common/scripts/upload-swift-artifacts</i></b></a><a href="https://github.com/openstack/tripleo-common/blob/master/scripts/upload-puppet-modules" target="_blank"> </a>and a slightly modified version which optimizes the flow for deploying directories containing puppet modules called <a href="https://www.blogger.com/"><b><i><span id="goog_392944846"></span>tripleo-common/scripts/upload-puppet-modules</i></b></a><span id="goog_392944847"></span><br />To make using these easier, I append this to my .bashrc<br />
<br /><br /><span style="font-family: "Courier New",Courier,monospace;">export PATH="$PATH:/home/stack/tripleo-common/scripts"</span><br />
<h3>
</h3>
<h3>
Example 1 - Deploy Artifacts "Hello World"</h3>
<br />So, let's start with a really simple example. First lets create a tarball containing a single /tmp/hello file:<br /><br /><span style="font-family: "Courier New",Courier,monospace;"><br />[stack@instack ~]$ mkdir tmp<br />[stack@instack ~]$ echo "hello" > tmp/hello<br />[stack@instack ~]$ tar -cvzf hello.tgz tmp<br />tmp/<br />tmp/hello</span><br /><br /><br />Now, we simply run the <b><i>upload-swift-artifacts</i></b> script, accepting all the default options other than to pass a reference to <b><i>hello.tgz</i></b><br /><br /><br /><span style="font-family: "Courier New",Courier,monospace;">[stack@instack ~]$ upload-swift-artifacts -f hello.tgz<br />Creating heat environment file: /home/stack/.tripleo/environments/deployment-artifacts.yaml<br />Uploading file to swift: hello.tgz<br />hello.tgz<br />Upload complete.<br /></span><br />There are currently only two supported file types:<br /><br /><ul>
<li> A tarball (will be unpacked from / on all nodes)</li>
<li> An RPM file (will be installed on all nodes) </li>
</ul>
<br />Taking a look inside the environment file the script generated, we can see it's using the DeployArtifactURLs parameter, and passing a single URL (the parameter accepts a list of URLs). This happens to be a swift tempurl, created by the <b><i>upload-swift-artifacts</i></b> script but it could be any URL accessible to the overcloud nodes at deployment time.<br /><span style="font-family: "Courier New",Courier,monospace;"><br />[stack@instack ~]$ cat /home/stack/.tripleo/environments/deployment-artifacts.yaml<br /># Heat environment to deploy artifacts via Swift Temp URL(s)<br />parameter_defaults:<br /> DeployArtifactURLs:<br /> - 'http://192.0.2.1:8080/v1/AUTH_e9bcd2a11af94c319b164eba73c59a28/overcloud/hello.tgz?temp_url_sig=96ae277d85c3ee38dd61234b8c99351e64c8bd45&temp_url_expires=1502273853'</span><br /><br />This environment file is automatically generated by the <b><i>upload-swift-artifacts</i></b> script, and put into the special <b><i>~/.tripleo/environments</i></b> directory. This directory is read by tripleoclient and any environment files included here are always included automatically (no need for any -e options), but you can also pass a --environment option to upload-swift-artifacts if you prefer some different output location (e.g so it can be explicitly included in your overcloud deploy command).<br /><br />Testing this example, you simply do an overcloud deployment, no additional arguments are needed if you use the default <b><i>.tripleo/environments/deployment-artifacts.yaml</i></b> environment path:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">[stack@instack ~]$ openstack overcloud deploy --templates</span><br /><br />Then check on one of the nodes for the expected file (note the tarball is unpacked from / in the filesystem):<br /><br /><span style="font-family: "Courier New",Courier,monospace;">[root@overcloud-controller-0 ~]# cat /tmp/hello<br />hello</span><br />Note the deploy artifact files are written to <b>all roles</b>, currently there is no way to deploy e.g only to Controller nodes. We might consider an enhancement that allows role specific artifact URL parameters in future should folks require it.<br />
<br />
Hopefully despite the very simple example you can see that this is a very flexible interface - you can deploy a tarball containing anything, e.g even configuration files such as policy.json files to the nodes.<br />
<br />
Note that you have to be careful though - most service configuration files are managed by puppet, so if you attempt using the deploy artifacts interface to overwrite puppet managed files it will not work - puppet runs after deploy artifacts are created (this is deliberate, as you will see in the next example) so you must use puppet hieradata to influence any configuration managed by puppet. (In the case of policy.json files, there is a <a href="https://github.com/openstack/puppet-openstacklib/blob/master/manifests/policy.pp" target="_blank">puppet module</a> that handles this, but currently TripleO does not use it - this may change in future though).<br /><br />
<h3>
Example 2 - Puppet development workflow</h3>
There is coupling between tripleo-heat-templates and the puppet modules it interfaces with (and in particular with the puppet profiles that exist in puppet-tripleo, as discussed in my composable services tutorial recently), so a common pattern for a developer is:<br />
<br />
<ol>
<li>Modify some puppet code</li>
<li>Modify tripleo-heat-templates to match the new/modified puppet profile</li>
<li>Deploy an overcloud</li>
<li><b>*OH NO* it doesn't work!</b></li>
<li>Debug the issue (hint, <i>"<b>openstack stack failures list overcloud</b>"</i><b> </b>is a super-useful new heatclient command which helps a lot here, as it surfaces the puppet error in most cases)</li>
<li>Make coffee; goto (1) :)<b> </b></li>
</ol>
Traditionally for TripleO deployments all puppet modules (including the <a href="https://github.com/openstack/puppet-tripleo/" target="_blank">puppet-tripleo</a> profiles) have been built into the image we deploy (stored in Glance on the undercloud), so one missing step above is getting the modified puppet code into the image. There are a few options:<br />
<br />
<ul>
<li>Rebuild the image every time (this is really slow)</li>
<li>Use virt-customize or virt-copy-in to copy some modifications into the image, then update the image in glance (this is faster, but it still means you must redeploy the nodes every time and it's easy to lose track of what modifications have been made).</li>
<li>Use <b><i>DeployArtifactUrls</i></b> to update the puppet modules on the fly during the deployment!</li>
</ul>
This last use-case is actually what prompted implementation of the DeployArtifacts interface (thanks <a href="https://dprince.github.io/" target="_blank">Dan</a>!), and I'll show how it works below:<br />
<br />
<span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z">First, we clone one or more puppet modules to a local directory - note the name of the repo e.g "puppet-tripleo" <b>does not</b> match the name of the deployed directory (on the nodes it's /etc/puppet/modules/tripleo), so you have to clone it to the "tripleo" directory.</span><br />
<span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;"><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z">mkdir puppet-modules</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z">cd puppet-modules</span><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z"> </span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z">git clone </span><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z url"><a href="https://git.openstack.org/openstack/puppet-tripleo">https://git.openstack.org/openstack/puppet-tripleo</a></span><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z"> tripleo</span></span><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z"> </span><br />
<br />
<span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z">Now you can make whatever edits are needed, pull under review code (or just do nothing if you want to deploy latest trunk of a given module). When you're ready you run the <b><i>upload-puppet-modules</i></b> script:</span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z">upload-puppet-modules -d puppet-modules</span></span><br />
<br />
This works a little bit differently to the previous upload-swift-artifacts script, it takes the directory, creates a tarball using the --transform option, so we rewrite the prefix from <i>/somewhere/puppet-modules</i> to <i>/etc/puppet/modules</i><br />
<br />
The process after we create the tarball is exactly the same - we upload it to swift, get a tempurl, and create a heat environment file which references the location of the tarball. On deployment, the updated puppet modules will be untarred and this always happens before puppet runs, which makes the debug workflow above much faster, nice!<br />
<i> </i><br />
<i><b>NOTE:</b> </i>There is one gotcha here - <span class="author-a-z71zlez84ztjbaj3z71zrz75zjz122zz75z"><b>upload-puppet-modules</b> creates a differently named environment file ($HOME/.tripleo/environments/puppet-modules-url.yaml) to <b>upload-swift-artifacts</b> by default, and their content is conflicting - if both environment files exist, one will be ignored as they will get merged together. (This is something we can probably improve in future when this <a href="https://blueprints.launchpad.net/heat/+spec/environment-merging" target="_blank">heat feature</a> lands, but right now the only option is to stick to one script or the other, or accept manually merging the environment files (to append rather than overwrite the DeployArtifactUrls parameter)<i><br /></i></span><br />
<br />
<br />
<br />
<h2>
So how does it work?</h2>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYBPzOeuFIAUL3uZ275H8n5iuy75etYeY1y7SS0Yt4KEjTOoxsCO2tZs9sZqcuZA2jusECaLYvy_596Cv3B2ACziUwK6HUHK77zhI_wT2UfWb7dQvkwJtnbQdrvTV_mAiceFw7NfxnHw8/s1600/DeployArtifacts2%25281%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYBPzOeuFIAUL3uZ275H8n5iuy75etYeY1y7SS0Yt4KEjTOoxsCO2tZs9sZqcuZA2jusECaLYvy_596Cv3B2ACziUwK6HUHK77zhI_wT2UfWb7dQvkwJtnbQdrvTV_mAiceFw7NfxnHw8/s640/DeployArtifacts2%25281%2529.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Deploy Artifacts Overview</td></tr>
</tbody></table>
<br />So, it's actually pretty simple, as illustrated in the diagram above<br />
<br />
<ul>
<li>A tarball is created containing the files you want to deploy to the nodes</li>
<li>This tarball is uploaded to swift on the undercloud</li>
<li>A Swift tempurl is created, so the tarball can be accessed using a signed URL (no credentials needed in the nodes to access)</li>
<li>A Heat environment passes the Swift tempurl to a nested stack "deploy-artifacts.yaml", which defines a DeployArtifactUrls parameter (which is a list)</li>
<li>deploy-artifacts.yaml defines a Heat SoftwareConfig resource, which references a shell script that can download files from a list of URLs, check the file type and do something (e.g in the case of a tarball, untar it!)</li>
<li>The deploy-artifacts SoftwareConfig is deployed inside the per-role "PostDeploy" template, which is where we perform the puppet steps (5 deployment passes which apply puppet in a series of steps). </li>
<li>We use the heat depends_on directive to ensure that the DeployArtifacts deployment (ControllerArtifactsDeploy in the case of the Controller role) always runs before any of the puppet steps.</li>
<li>This pattern is replicated for all roles (not just the Controller as in the diagram above) </li>
</ul>
As you can see, there are a few steps to the process, but it's pretty simple and it leverages the exact same <a href="http://hardysteven.blogspot.co.uk/2015/05/heat-softwareconfig-resources.html" target="_blank">Heat SoftwareDeployment patterns</a> we use throughout TripleO to deploy scripts (and apply puppet manifests, etc).Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com6tag:blogger.com,1999:blog-7962934533489004582.post-13767984140363049392016-08-05T07:02:00.000-07:002016-10-10T08:59:53.635-07:00TripleO Composable Services 101Over the newton cycle, we've been working very hard on a major refactor of our heat templates and puppet manifiests, such that a much more granular and flexible "Composable Services" pattern is followed throughout our implementation.<br />
<br />
It's been <a href="https://blueprints.launchpad.net/tripleo/+spec/composable-services-within-roles" target="_blank">a lot of work</a>, but it's been a frequently requested feature for some time, so I'm excited to be in a position to say it's complete for Newton (kudos to everyone involved in making that happen!) :)<br />
<br />
This post aims to provide an introduction to this work, an overview of how it works under the hood, some simple usage examples and a roadmap for some related follow-on work.<br />
<br />
<br />
<a name='more'></a><br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Why Composable Services?</span></h2>
<br />
It probably helps to start with some historical context here. As described in <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-1-roles-and.html" target="_blank">previous posts</a> TripleO has provided a fixed architecture with 5 roles (where "roles" means groups of nodes) Controller, Compute, BlockStorage, CephStorage and ObjectStorage.<br />
<br />
To configure each of these roles, we used puppet, and we had a l<a href="https://github.com/openstack/tripleo-heat-templates/tree/stable/mitaka/puppet/manifests" target="_blank">arge manifest per role</a>, with some relatively inflexible assumptions about which services would run on each role.<br />
<br />
This worked OK, but many users have been requesting more flexibility, such as:<br />
<br />
<ul>
<li>Ability to easily disable services they don't need</li>
<li>Allow service placement choice, such as co-locating the Ceph OSD service with nova-compute services to reduce the required hardware footprint (so-called "hyperconverged" deployments)</li>
<li>Make it easier to integrate new services and integrate third-party pieces (get closer to a strongly defined "plugin" interface)</li>
</ul>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-MztAf4UXKS04XwL9S6vlyUMOHk_CjFr9GsX6eC0MJKw_RKUWYGiyyExYsBeguyEBglg0VJ8RA-bUpQEyCAnX15jbEUNL55eP3YFkGTlPtRA0OjlGP7ZAKyo1lUhgG9Tvrri-Hnjgqy4/s1600/OvercloudControllerPuppet+%25281%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="465" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-MztAf4UXKS04XwL9S6vlyUMOHk_CjFr9GsX6eC0MJKw_RKUWYGiyyExYsBeguyEBglg0VJ8RA-bUpQEyCAnX15jbEUNL55eP3YFkGTlPtRA0OjlGP7ZAKyo1lUhgG9Tvrri-Hnjgqy4/s640/OvercloudControllerPuppet+%25281%2529.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The pre-newton Tripleo architecture, one manifest and heat template per role.</td></tr>
</tbody></table>
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">So, how does it work?</span></h2>
So, basically we've made two fundamental changes to our interfaces:<br />
<br />
<ul>
<li>Each service, e.g "nova-api" is now defined by an <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml" target="_blank">individual heat template.</a> The interfaces for these are standardized so all services must implement a basic subset of input parameters and output values.</li>
<li>Every service defines a small <a href="https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/nova/api.pp" target="_blank">puppet "profile"</a>, which is a puppet manifest fragment that defines configuring that service. Again a standard interface is used, in particular a "step" variable is passed to each puppet profile, so you can choose which step configuration occurs in (we apply configuration in a <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/README.rst#steps" target="_blank">series of six steps</a> so the author of the profile can choose when a service is configured relative to other services).</li>
</ul>
This is the basis of the TripleO "service plugin" interface, and it should enable *much* easier integration of new services, and hopefully provide a more accessible interface to new contributors.<br />
<br />
Inside the TripleO templates, we made use of a new-for-mitaka <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::ResourceChain" target="_blank">Heat ResourceChain</a> interface to compose a deployment of multiple services. Basically a ResourceChain is a group of resources that may have different types, but conform to the same interfaces, which is what we need to combine a bunch of service templates that all have some standard interfaces.<br />
<br />
Here's an illustration of how it works - essentially you define an input parameter which is a list of services, e.g <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.yaml#L173" target="_blank">OS::TripleO::Services: NovaApi</a> which then maps to the heat template for that service, e.g <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml" target="_blank">puppet/services/nova-api.yaml</a> via the <a href="http://docs.openstack.org/developer/heat/template_guide/environment.html" target="_blank">resource_registry interface</a> discussed in <a href="http://hardysteven.blogspot.co.uk/2013/10/heat-providersenvironments-101-ive.html" target="_blank">previous posts</a>. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibapVLtWr1omI6-Kmx9ISN-rCkQzsUfuCMgttusvwwnYUAy0JUlSiHP0djxH-DZiqIIfd5SGI2RC3N8coHDxOFWJiXsTJFxM6jExc4PGr-0rnpqekyMXikrdRUBR_qsTggFiQHCqQLcoE/s1600/Copy+of+OvercloudControllerPuppetComposable+%25281%2529.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="544" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibapVLtWr1omI6-Kmx9ISN-rCkQzsUfuCMgttusvwwnYUAy0JUlSiHP0djxH-DZiqIIfd5SGI2RC3N8coHDxOFWJiXsTJFxM6jExc4PGr-0rnpqekyMXikrdRUBR_qsTggFiQHCqQLcoE/s640/Copy+of+OvercloudControllerPuppetComposable+%25281%2529.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">For Newton, each role has a ServiceChain that combines the chosen services for that role.</td></tr>
</tbody></table>
<br />
If you'd like more information on the implementation details, I'd encourage you to check out the <a href="http://docs.openstack.org/developer/tripleo-docs/tht_walkthrough/tht_walkthrough.html" target="_blank">developer documentation</a> where we're starting to document these interfaces in more detail.<br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Ok, how do I use it?</span></h2>
<h2>
</h2>
Here I'm going to focus on usage of the feature vs developing new services (which is pretty well covered in the aforementioned <a href="http://docs.openstack.org/developer/tripleo-docs/tht_walkthrough/tht_walkthrough.html" target="_blank">developer docs</a>), and hopefully illustrate why this is an important step forward that improves operator deployment choices.<br />
<br />
<h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Scenario 1 - All in one minimal deployment</span></h3>
Lets say for a moment that you're a keystone developer and you want a shorter debug cycle and/or are resource constrained. With the new interfaces, it's become very easy to deploy a minimal subset of services on a single node:<br />
<br />
First you create an environment file that overrides the default <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L356" target="_blank">ControllerServices list</a> (which at the time of writing contains about 50 services!) so it only includes <span style="font-family: "courier new" , "courier" , monospace;">OS::TripleO::Services::Keystone and the services keystone depends on. We also set ComputeCount to zero as we don't need any compute nodes.</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat keystone-only.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">parameter_defaults:<br /> ControllerServices:<br /> - OS::TripleO::Services::Keystone<br /> - OS::TripleO::Services::RabbitMQ</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - OS::TripleO::Services::HAproxy<br /> - OS::TripleO::Services::MySQL<br /> ComputeCount: 0</span><br />
<br />
<i>(Note that in some environments it may also be necessary to include the OS::TripleO::Services::Pacemaker too)</i><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">You can then deploy your single node keystone-only environment:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">openstack overcloud deploy --templates -e keystone_only.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">When this completes, you'll see the following message, and you can source the overcloudrc and get a token to prove the deployed keystone is working:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">...</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Overcloud Endpoint: http://192.0.2.15:5000/v2.0<br />Overcloud Deployed<br />[stack@instack ~]$ . overcloudrc <br />[stack@instack ~]$ openstack token issue<br />+------------+----------------------------------+<br />| Field | Value |<br />+------------+----------------------------------+<br />| expires | 2016-08-05 10:16:16+00:00 |<br />| id | 976d5fcf9f744a5a9cf840e83d825560 |<br />| project_id | 99e92ae58d1f4147a5d7eda0af516060 |<br />| user_id | 29fe578e45b24406ba6c5fd0baaeaa9c |<br />+------------+----------------------------------+</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">We can see by looking at the undercloud nova (don't forget to source the stackrc after interacting with the overcloud above!) that there is one controller node):</span><br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[stack@instack ~]$ . stackrc <br />[stack@instack ~]$ nova list<br />+--------------------------------------+------------------------+--------+------------+-------------+--------------------+<br />| ID | Name | Status | Task State | Power State | Networks |<br />+--------------------------------------+------------------------+--------+------------+-------------+--------------------+<br />| d5155616-d2a6-4cee-a6d1-37bb83fccfe0 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.7 |<br />+--------------------------------------+------------------------+--------+------------+-------------+--------------------+</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<h3>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Scenario 2 - "hyperconverged" ceph deployment</span></h3>
In this case, we want to move the Ceph OSD services, which normally run on the CephStorage role, and instead have them run on the Compute role.<br />
<br />
To do this, we first look at the default values for the ComputeServices and CephStorageServices parameters in <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L414">overcloud.yaml</a> (as in the example above for the Controller role, these lists define the services to be deployed on the Compute and CephStorage roles respectively):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L414">ComputeServices</a>:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> default:<br /> - OS::TripleO::Services::CephClient<br /> - OS::TripleO::Services::CephExternal<br /> - OS::TripleO::Services::Timezone<br /> - OS::TripleO::Services::Ntp<br /> - OS::TripleO::Services::Snmp<br /> - OS::TripleO::Services::NovaCompute<br /> - OS::TripleO::Services::NovaLibvirt<br /> - OS::TripleO::Services::Kernel<br /> - OS::TripleO::Services::ComputeNeutronCorePlugin<br /> - OS::TripleO::Services::ComputeNeutronOvsAgent<br /> - OS::TripleO::Services::ComputeCeilometerAgent</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L528">CephStorageServices</a>:<br /> default:<br /> - OS::TripleO::Services::CephOSD<br /> - OS::TripleO::Services::Kernel<br /> - OS::TripleO::Services::Ntp<br /> - OS::TripleO::Services::Timezone</span><br />
<br />
<br />
Our aim is to deploy one Compute node, running both the standard compute services, and the OS::TripleO::Services::CephOSD service (the other services are clearly common to both roles). We also don't need the OS::TripleO::Services::CephExternal service defined in ComputeServices, because we won't be referencing any external ceph cluster, which gives us this:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat ceph_osd_on_compute.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">parameter_defaults:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ComputeServices:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - OS::TripleO::Services::CephClient<br /><b> - OS::TripleO::Services::CephOSD</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - OS::TripleO::Services::Timezone<br /> - OS::TripleO::Services::Ntp<br /> - OS::TripleO::Services::Snmp<br /> - OS::TripleO::Services::NovaCompute<br /> - OS::TripleO::Services::NovaLibvirt<br /> - OS::TripleO::Services::Kernel<br /> - OS::TripleO::Services::ComputeNeutronCorePlugin<br /> - OS::TripleO::Services::ComputeNeutronOvsAgent<br /> - OS::TripleO::Services::ComputeCeilometerAgent </span><br />
<br />
That is all that's required to enable a hyperconverged ceph deployment! :)<br />
<br />
Since the default count for CephStorage is zero, we can then deploy like this:<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[stack@instack ~]$ openstack overcloud deploy --templates /tmp/tripleo-heat-templates -e ceph_osd_on_compute.yaml -e /tmp/tripleo-heat-templates/environments/storage-environment.yaml</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Here we can see I'm specifying a non-default location <i>/tmp/tripleo-heat-templates</i> for the template tree (this defaults to <i>/usr/share/openstack-tripleo-heat-templates</i>), passing the <i>ceph_osd_on_compute.yaml</i> environment to enable the OSD service on the Compute role, and finally passing the storage-environment.yaml that configures things so they are backed by Ceph.</span></span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Logging onto the compute node after deployment we see this:</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace;">[root@overcloud-novacompute-0 ~]# ps ax | grep ceph<br />17437 ? Ss 0:00 /bin/bash -c ulimit -n 32768; /usr/bin/ceph-osd -i 0 --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf --cluster ceph -f<br />17438 ? Sl 0:00 /usr/bin/ceph-osd -i 0 --pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf --cluster ceph -f</span></span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">So, it worked, and we have the OSD service running on the Compute role! :)</span></span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Similar patterns to those described above can be used to achieve various deployment topologies which were not previously possible (an all-in-one deployment including nova-compute on a single node for example, as is done in <a href="https://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/multinode.yaml#L6">one of our CI jobs</a> now)</span></span><br />
<br />
<h2>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Future Work</span></span></h2>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Hopefully by now you can see that these new interfaces provide a much cleaner abstraction for services, and a lot more operator flexibility regarding their placement. However for some environments this is not enough, and completely new roles may be needed. We're working towards enabling that via the <a href="https://blueprints.launchpad.net/tripleo/+spec/custom-roles">custom-roles blueprint</a>, which will hopefully <a href="https://launchpad.net/tripleo/+milestone/newton-3">land for Newton.</a></span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Another related piece of work is <a href="https://review.openstack.org/#/c/330414/">enabling more flexible environment merging inside Heat.</a> This will mean there is less need to specify the full list of Services as described above, and instead we'll be able to build up a list of services based on multiple environment files (which are then merged appending to the final list).</span></span><br />
<h3>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></h3>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com4tag:blogger.com,1999:blog-7962934533489004582.post-85208817130551399162016-06-09T04:59:00.000-07:002016-06-09T04:59:31.379-07:00TripleO partial stack updatesRecently I was asked if it's possible to do a partial update of a TripleO overcloud - the answer is yes, so I thought I'd write a post showing how to do it. Much of what follows is basically an update on my <a href="http://hardysteven.blogspot.co.uk/2013/08/heat-nested-resource-introspection.html" target="_blank">old post on nested resource introspection</a> (some interfaces have changed a bit since I wrote that), combined with an introduction to heat PATCH updates.<br />
<br />
<h2>
Partial update?! Why?</h2>
So, the first question is why would you do this - TripleO heat templates are designed to enforce a consistent state for your entire OpenStack deployment, so in most cases you really should update the entire overcloud, and not mess with the underlying nested stacks directly.<br />
<br />
However, for some development usage, this creates a long feedback loop - you change something (perhaps one line in a puppet manifest or heat template), then have to wait several minutes for Heat to walk the entire tree of nested stacks, puppet to run all steps on all nodes, etc.<br />
<br />
So, while you would probably never do this in production (seriously, please don't!), it can be a useful technique for developers seeking a quicker hack-then-test cycle, and also when attempting to isolate root-causes for some subset of overcloud stack update behavior.<br />
<br />
Ok, with that disclaimer clearly stated, here's how you do it:<br />
<br />
<h2>
Step 1 - Find the nested stack to update</h2>
<br />Lets take a specific example - I want to update only the <a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/mitaka/overcloud.yaml#L1635" target="_blank">ControllerNodesPostDeployment resource</a> which is defined in <a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/mitaka/overcloud.yaml#L1635" target="_blank">overcloud.yaml </a>- this is a resource that <a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/mitaka/overcloud-resource-registry-puppet.yaml#L13" target="_blank">maps to a nested stack</a> that uses the cluster configuration interfaces <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html" target="_blank">I described in this previous post</a> to apply puppet in a series of steps to all controller nodes.<br />
<br />
Here's our overcloud (some CLI output removed for brevity):<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat stack-list<br />| 01c51e7e-ad2f-41d3-b056-3c4c84395114 | overcloud | CREATE_COMPLETE |<br />2016-06-08T18:07:00 | None |</span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: inherit;">Here's the ControllerNodesPostDeployment resource:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat resource-list overcloud | grep ControllerNodesPost<br />| ControllerNodesPostDeployment |<br /><b>e67fff24-8089-4cf8-adf4-9c6064bf01d6</b> |<br />OS::TripleO::ControllerPostDeployment | CREATE_COMPLETE |<br />2016-06-08T18:07:00 |<br /></span><b>e67fff24-8089-4cf8-adf4-9c6064bf01d6</b> is the resource ID of<br />ControllerNodesPostDeployment, which is a nested stack - you can confirm<br />this via:<br /><br /><span style="font-family: "Courier New",Courier,monospace;">$ heat stack-list -n | grep "^| <b>e67fff24-8089-4cf8-adf4-9c6064bf01d6</b>"<br />| <b>e67fff24-8089-4cf8-adf4-9c6064bf01d6</b> |<br />overcloud-ControllerNodesPostDeployment-smy5ygz2lc26<br />| UPDATE_COMPLETE | 2016-06-08T18:10:34 | 2016-06-09T08:52:45 |<br />01c51e7e-ad2f-41d3-b056-3c4c84395114 |</span><br />Note here the first column is the stack ID, and the last is the parent<br />stack ID (e.g "overcloud" above).<br />
<br />
<b>overcloud-ControllerNodesPostDeployment-smy5ygz2lc26</b> is the name of the stack that implements ControllerNodesPostDeployment - we can refer to it by either that name or the ID (<b>e67fff24-8089-4cf8-adf4-9c6064bf01d6</b>).<br />
<br />
<h2>
Step 2 - Basic update of the stack</h2>
Heat supports PATCH updates, so it is possible to trigger a no-op update without passing any template or parameters (the existing data will be used), or to patch in some specific modification.<br />
<br />
Here's now it works, we simply use either the name or ID we discovered above, and use heat stack-update (or the new openstack client equivalent commands.<br />
<br />
First, however, we want to get the last event ID before triggering the update (or, on recent heatclient versions you can instead use openstack stack event list --follow):<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat event-list overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 | tac | head -n2<br />+------------------------------------------------------+--------------------------------------+-------------------------------------+--------------------+---------------------+<br />| overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 | <b>89e535ef-d414-4121-b726-9924eccb4fc3</b> | Stack UPDATE completed successfully | UPDATE_COMPLETE | 2016-06-09T09:09:09 |<br /></span><br />
<br />
So the last event logged by this nested stack has the ID of <b>89e535ef-d414-4121-b726-9924eccb4fc3 </b>- we can use this as a marker so we hide all previous events for the stack:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ heat event-list -m 89e535ef-d414-4121-b726-9924eccb4fc3 overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 <br />+----+------------------------+-----------------+------------+<br />| id | resource_status_reason | resource_status | event_time |<br />+----+------------------------+-----------------+------------+<br />+----+------------------------+-----------------+------------+</span><br />
Now, we can trigger the update, and use the marker event-list to follow progress:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">heat stack-update -x overcloud-ControllerNodesPostDeployment-smy5ygz2lc26</span><br />
<br />
<wait a short time><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat event-list -m 89e535ef-d414-4121-b726-9924eccb4fc3 overcloud-ControllerNodesPostDeployment-smy5ygz2lc26<br />+------------------------------------------------------+<br />| resource_name | id | resource_status_reason | resource_status | event_time |<br />+------------------------------------------------------+<br />| overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 | 2e08a022-ce0a-4e57-bf30-719fea6cbb74 | Stack UPDATE started | UPDATE_IN_PROGRESS | 2016-06-09T10:00:52 |<br />| ControllerArtifactsConfig | a55f9b17-f26c-4664-9ea5-535949c368e8 | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:00 |<br />| ControllerPuppetConfig | 21679c7f-c354-4319-9688-7fa290168664 | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:00 |<br />| ControllerPuppetConfig | f5761452-91dd-45dc-92e8-a5c371fa5004 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:02 |<br />| ControllerArtifactsConfig | 01abec3c-f472-4ec2-893d-0fddb8fc1696 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:02 |<br />| ControllerArtifactsDeploy | f8f7a21f-9169-4f8c-ab46-46ecbb141be8 | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:02 |<br />| ControllerArtifactsDeploy | 75937a57-e2f0-4d66-9b4c-2308593e56b1 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:04 |<br />| ControllerLoadBalancerDeployment_Step1 | 6058e29c-cded-4ad3-94d9-65909fd4911d | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:04 |<br />| ControllerLoadBalancerDeployment_Step1 | c9f93f1f-177c-4721-827f-a7d409b2cd50 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:06 |<br />| ControllerServicesBaseDeployment_Step2 | 92409e4c-24f2-4e68-bad9-47ce09107d7a | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:06 |<br />| ControllerServicesBaseDeployment_Step2 | a9203aa1-c438-47c0-977b-8e34669777bc | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:08 |<br />| ControllerOvercloudServicesDeployment_Step3 | aa7d78dc-d243-4d54-8ea6-3b59a6ed302a | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:08 |<br />| ControllerOvercloudServicesDeployment_Step3 | 4a1a6885-29d7-4708-a884-01f481ac1b35 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:10 |<br />| ControllerOvercloudServicesDeployment_Step4 | 7afd52c1-cbbc-431a-a22c-dd7459ed2255 | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:10 |<br />| ControllerOvercloudServicesDeployment_Step4 | 0dac2e72-0919-4e91-ac94-100d8d811c67 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:13 |<br />| ControllerOvercloudServicesDeployment_Step5 | ec57867f-e401-4756-bd30-0a566eced343 | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:13 |<br />| ControllerOvercloudServicesDeployment_Step5 | 427582fb-acd1-4939-a13c-7b3cbbc7527b | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:15 |<br />| ExtraConfig | 760fd961-fff6-4f4c-848e-80773e09e04b | state changed | UPDATE_IN_PROGRESS | 2016-06-09T10:01:15 |<br />| ExtraConfig | caee58b6-01bb-4805-b41f-4c48a8c7d767 | state changed | UPDATE_COMPLETE | 2016-06-09T10:01:16 |<br />| overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 | 35f527a5-0761-46bb-aecb-6eee0e0f083e | Stack UPDATE completed successfully | UPDATE_COMPLETE | 2016-06-09T10:01:25 |</span><br /><br />
<span style="font-family: inherit;">So, we can see that we triggered an update on the nested stack, and it ran to completion in around 30 seconds (much less time than updating the entire overcloud).</span><br />
<br />
<h2>
Step 3 - Update of the stack with modifications</h2>
So, those paying attention may have noticed that 30 seconds is too fast for puppet to run on all the controller nodes, and it is - the reason being that we did a no-op update, and so Heat detects that no inputs have changed, thus it doesn't cause puppet to re-run.<br />
<br />
To work around this, and enable puppet to re-assert state on every overcloud update, we have an <a href="https://github.com/openstack/tripleo-heat-templates/blob/stable/mitaka/puppet/controller-post.yaml#L13" target="_blank">identifier in the nested stack</a> that is normally updated to a value that changes every update (in includes a timestamp when updates are triggered via python-tripleoclient vs heatclient directly)<br />
<br />
We can emulate this behavior in our patch update, and force puppet to re-run through all the deployment steps - lets first look at the NodeConfigIdentifers parameter value:<br />
<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: small;">$ heat stack-show overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 | grep NodeConfigIdentifiers</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: small;">"NodeConfigIdentifiers": "{u'deployment_identifier': u'<b>1465409217</b>', u'controller_config': {u'0': u'os-apply-config deployment bb67a1d5-f0a5-48ec-9883-1f2ae578a8bd complet
ed,Root CA cert injection not enabled.,TLS not enabled.,None,'}, u'allnodes_extra': u'none'}" </span></span><br />
<br />
Here we can see various data, including a deployment_identifier, which is the timestamp-derived unique identifier normally passed via python-tripleoclient.<br />
<br />
We could update just that field, but the content of this mapping isn't important, only that it changes (this data is not currently consumed by puppet on update, it's just used to trigger the SoftwareDeployment to re-apply the config due to an input value changing).<br />
<br />
So we can create an environment file that looks like this (note this must use parameters, not parameter_defaults, so that it overrides the value passed from the parent stack) - any value can be used, but you must change it each update if you want the SoftwareDeployment resources to be re-applied to the nodes.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ cat update_env.yaml <br />parameters:<br /> NodeConfigIdentifiers: 123</span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<br />
Then we can trigger another PATCH update including this data:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">heat stack-update -x overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 -e update_env.yaml</span><br />
<br />
<span style="font-family: inherit;">This time I'm using the new openstack stack event list --follow approach to monitor progress (if you don't have this, you can repeat the marker event-list approach described above):</span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ openstack stack event list --follow</span><span style="font-family: "Courier New",Courier,monospace;">2016-06-09 08:52:46 [overcloud-ControllerNodesPostDeployment-smy5ygz2lc26]: UPDATE_IN_PROGRESS Stack UPDATE started<br />2016-06-09 08:52:54 [ControllerPuppetConfig]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:52:54 [ControllerArtifactsConfig]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:52:56 [ControllerPuppetConfig]: UPDATE_COMPLETE state changed<br />2016-06-09 08:52:56 [ControllerArtifactsConfig]: UPDATE_COMPLETE state changed<br />2016-06-09 08:52:56 [ControllerArtifactsDeploy]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:52:58 [ControllerArtifactsDeploy]: UPDATE_COMPLETE state changed<br />2016-06-09 08:52:58 [ControllerLoadBalancerDeployment_Step1]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:53:32 [ControllerLoadBalancerDeployment_Step1]: UPDATE_COMPLETE state changed<br />2016-06-09 08:53:32 [ControllerServicesBaseDeployment_Step2]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:54:00 [ControllerServicesBaseDeployment_Step2]: UPDATE_COMPLETE state changed<br />2016-06-09 08:54:00 [ControllerOvercloudServicesDeployment_Step3]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:54:57 [ControllerOvercloudServicesDeployment_Step3]: UPDATE_COMPLETE state changed<br />2016-06-09 08:54:57 [ControllerOvercloudServicesDeployment_Step4]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:56:14 [ControllerOvercloudServicesDeployment_Step4]: UPDATE_COMPLETE state changed<br />2016-06-09 08:56:14 [ControllerOvercloudServicesDeployment_Step5]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:57:16 [ControllerOvercloudServicesDeployment_Step5]: UPDATE_COMPLETE state changed<br />2016-06-09 08:57:16 [ExtraConfig]: UPDATE_IN_PROGRESS state changed<br />2016-06-09 08:57:17 [ExtraConfig]: UPDATE_COMPLETE state changed<br />2016-06-09 08:57:26 [overcloud-ControllerNodesPostDeployment-smy5ygz2lc26]: UPDATE_COMPLETE Stack UPDATE completed successfully</span><br />
<span style="font-family: inherit;"><span style="font-family: inherit;">So, here we can see the update of the stack too<span style="font-family: inherit;">k a little longer (around 5 minutes in my environment), and if you were to check the os-collect-config logs on each controller node, you would see puppet re-applying on each node, fore every step<span style="font-family: inherit;"> defined in the template<span style="font-family: inherit;">.</span></span></span></span></span><br />
<br />
<span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;">This approach can be extended if you <span style="font-family: inherit;">want to e.g test changes to the stack template (or files i<span style="font-family: inherit;">t references such as puppet manifests or scripts), y<span style="font-family: inherit;">ou would do something like:</span></span></span></span></span></span></span></span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ cp -r /usr/share/openstack-tripleo-heat-templates .</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ cd openstack-tripleo-heat-templates/</span><br />
<span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">$ heat stack-update -x overcloud-ControllerNodesPostDeployment-smy5ygz2lc26 -e update_env.yaml -f puppet/controller-post.yaml</span></span></span></span></span></span></span></span></span><br />
<span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><br /></span></span></span></span></span></span></span></span>
<span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;"><span style="font-family: inherit;">Note that if you want to do a final update of the entire over<span style="font-family: inherit;">cl<span style="font-family: inherit;">oud, you would need to point to this copied t<span style="font-family: inherit;">ree (assuming you want to maintain any changes)<span style="font-family: inherit;">, e.g</span></span></span></span></span></span></span></span></span></span></span></span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ openstack overcloud deploy --templates /path/to/copy/openstack-tripleo-heat-templates</span> <br />
<br />
<span style="font-family: inherit;"><br /></span>Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com12tag:blogger.com,1999:blog-7962934533489004582.post-27481025677236006032015-05-17T01:06:00.003-07:002015-05-17T13:52:01.815-07:00TripleO Heat templates Part 3 - Cluster configuration, introduction/primerIn my previous two posts I covered an <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-1-roles-and.html" target="_blank">overview of TripleO template roles and groups</a>, and specifics of how <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-2-node.html" target="_blank">initial deployment of a node happens</a>. Today I'm planning to introduce the next step of the deployment process - taking the deployed groups of nodes, and configuring them to work together as clusters running the various OpenStack services encapsulated by each role.<br />
<br />
This post will provide an introduction to the patterns and Heat features used to configure the groups of nodes, then in the next instalment I'll dig into the specifics of exactly what configuration takes place in the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml" target="_blank">TripleO heat templates.</a><br />
<br />
<br />
<a name='more'></a><br />
<br />
<h2>
Recap - the deployed group of servers</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXYeKikplIRvvnxcYkKk_wa8BF5rlwnkAgJsWpIo0pvcnEkhILu5R5G0Cs3F1Uj6MzvDVTEFEmE9pz4Hl6sw9KUoNfsFFvX9UWScdm5a-nGuI6B8zHc3Wm5eKX4lmBZxQeBY4n9Vt0Am4/s1600/TripleOGroup2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXYeKikplIRvvnxcYkKk_wa8BF5rlwnkAgJsWpIo0pvcnEkhILu5R5G0Cs3F1Uj6MzvDVTEFEmE9pz4Hl6sw9KUoNfsFFvX9UWScdm5a-nGuI6B8zHc3Wm5eKX4lmBZxQeBY4n9Vt0Am4/s400/TripleOGroup2.png" width="400" /></a></div>
<br />
So, we're continuing from where we got to at the end of the <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-2-node.html" target="_blank">last post</a> - we've deployed a <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::ResourceGroup" target="_blank">ResourceGroup</a> containing several OS::TripleO::Controller resources, which in turn have deployed a nova server, and done some initial configuration of it. <br />
<br />
What comes next is configuring the whole group, or cluster, to work together, e.g configuring the OpenStack services running on the controller.<br />
<br />
<h2>
Group/Cluster configuration with Heat</h2>
<br />
Similar to the <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployment" target="_blank">SoftwareDeployment (singular)</a> resources described in my <a href="http://hardysteven.blogspot.co.uk/2015/05/heat-softwareconfig-resources.html" target="_blank">previous post</a>, Heat supports applying a SoftwareConfig to a group of servers via the <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployments" target="_blank">SoftwareDeployments</a> and <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::StructuredDeployments" target="_blank">StructuredDeployments</a> (plural) resources. The function of both is basically the same, one works with a <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareConfig" target="_blank">SoftwareConfig</a> resource and the other with a <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::StructuredConfig" target="_blank">StructuredConfig</a> resource.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPZnEtltJ6070R3DWjSy-Lt-qa3X2_NaIsl2jjd9GWloPtNS34voArgALlu3UGjhw0gm0cJSTCncQeT-ebTZ7ar3wAg4Xj30rs2tAgwjMWju4_4jpAAT-X1Xzj6J8wxcHBH5MFJ0XI3fY/s1600/SoftwareDeploymentsFlow2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="344" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPZnEtltJ6070R3DWjSy-Lt-qa3X2_NaIsl2jjd9GWloPtNS34voArgALlu3UGjhw0gm0cJSTCncQeT-ebTZ7ar3wAg4Xj30rs2tAgwjMWju4_4jpAAT-X1Xzj6J8wxcHBH5MFJ0XI3fY/s640/SoftwareDeploymentsFlow2.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Typically (in TripleO at least) StructuredDeployments resources are used combined with a
ResourceGroup containing some servers. You pass a list of servers to
configure (provided via an attribute from the OS::Heat::ResourceGroup
resource), and a reference to a StructuredConfig resource.<br />
<br />
The StructuredConfig resource defines the configuration to apply to
each server, and the StructuredDeployments resource then internally
creates a series of StructuredDeployment (singular) resources, one per
server.<br />
<br />
When all of the deployment (singular) resources complete, the deployments (plural) resource goes CREATE_COMPLETE - if any of the nested deployment resources fail, the deployments resource will go into a FAILED state. <br />
<br />
<h2>
Debugging groups of deployments</h2>
<br />
You may notice that the StructuredDeployments
resource above looks a lot like the ResourceGroup containing the OS::TripleO::Controller resources
- this is no coincidence, internally heat actually creates a
ResourceGroup containing the StructuredDeployment resources.<br />
<br />
This
is a useful fact to remember when debugging, because it means you can
use the <a href="http://hardysteven.blogspot.co.uk/2015/04/debugging-tripleo-heat-templates.html" target="_blank">techniques I've previously described</a> to inspect the individual Deployment resources
created by the StructuredDeployments resource, e.g so you can use heat deployment-show
<id> to help diagnose a problem with a failing deployment inside
the StructuredDeployments group<span style="font-family: inherit;"> (which is ofte<span style="font-family: inherit;">n quicker and more convenient than SSHing onto the failing node and trawling the logs).</span></span><span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"><br /></span></span><br />
<br />
For example, here's a simple bash script which dumps out details about all of the Deployment resources in an overcloud, obviously you can add in a "grep FAILED" here if you just want to see details about failing deployments:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">#!/bin/bash<br />while read -r line<br />do<br /> deployment_name=$(echo $line | cut -d"|" -f2)<br /> deployment_id=$(echo $line | cut -d"|" -f3)<br /> parent_name=$(echo $line | cut -d"|" -f7)<br /> echo "deployment=$deployment_name ($deployment_id) parent $parent_name"<br /> heat deployment-show $deployment_id<br /> echo "---"<br />done < <(heat resource-list --nested-depth 5 overcloud | grep "OS::Heat::\(Software\|Structured\)Deployment ")</span></span><br />
<br />
We should probably add a python-heatclient feature which automates this lookup (particularly for failing deployments), but right now that is one way to do it.<br />
<h2>
</h2>
<h2>
Until next time..! </h2>
So here we've covered the basics of how Heat can be used to configure groups of servers, and we've illustrated how that pattern is applied in the TripleO templates.<br />
<br />
The TripleO templates use this technique for all roles, to do multiple configuration passes during the deployment - in the next post I'll cover specifics of how this works in detail, but for now you can check out the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml" target="_blank">TripleO heat templates</a> and hopefully see this pattern for yourself. Note that it's combined with <a href="http://hardysteven.blogspot.co.uk/2013/10/heat-providersenvironments-101-ive.html" target="_blank">provider resource abstractions</a> as <a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-1-roles-and.html" target="_blank">previously discussed</a>, which as we will see makes for a nicely abstracted approach to cluster configuration which is pretty easy to modify, extend, or plug in alternative implementations.Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com19tag:blogger.com,1999:blog-7962934533489004582.post-49930047528517500202015-05-13T09:02:00.000-07:002015-05-13T09:02:59.102-07:00TripleO Heat templates Part 2 - Node initial deployment & configIn my previous post "<a href="http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-1-roles-and.html" target="_blank">TripleO Heat templates Part 1 - roles and groups</a>", I provided an overview of the various TripleO roles, the way the role implementation is abstracted via <a href="http://hardysteven.blogspot.co.uk/2013/10/heat-providersenvironments-101-ive.html" target="_blank">provider resources</a>, and how they are grouped and scaled via <a href="http://hardysteven.blogspot.co.uk/2014/09/using-heat-resourcegroup-resources.html" target="_blank">OS::Heat::ResourceGroup</a>.<br />
<br />
In this post, I'm aiming to dig into the next level of template implementation, specifically how a role is implemented behind the provider resource alias used in the top-level template.<br />
<br />
I'm only going to cover one role type for now <b>OS::TripleO::Controller</b>, because the patterns described are directly applicable to all other role types. I'm also going to focus on the puppet based implementation (because that's what I'm most familiar with), but again most concepts apply to the element/container/etc based implementations too.<br />
<br />
Throughout this post, I'll be referring to templates in the <a href="https://github.com/openstack/tripleo-heat-templates/" target="_blank">tripleo-heat-templates</a> repo, so if you haven't already, now might be a good time to clone that so you can follow along looking at the templates themselves.<br />
<br />
<a name='more'></a><br />
<h2>
Recap - the controller group definition</h2>
<h2>
</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvE04hyphenhyphen7Fs2erRUsoNevQAL3_tH3Xvb4HaDzO9GCKgOeebmgRuaxuAo4y7leOnn2BLRdK3RbK8ZRWviC9ovoyWZi1LmIQF_yxP41Mor3gTPa7dFLGyXtxFmCgddztinTJYpx66Yxuy-dQ/s1600/text12178-5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvE04hyphenhyphen7Fs2erRUsoNevQAL3_tH3Xvb4HaDzO9GCKgOeebmgRuaxuAo4y7leOnn2BLRdK3RbK8ZRWviC9ovoyWZi1LmIQF_yxP41Mor3gTPa7dFLGyXtxFmCgddztinTJYpx66Yxuy-dQ/s320/text12178-5.png" width="320" /></a></div>
<h2>
</h2>
So, as described in my previous post, the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml" target="_blank">top-level TripleO heat template </a>defines an OS::Heat::ResourceGroup called "Controller", which contains a group of OS::TripleO::Controller resources. <br />
<br />
This OS::TripleO::Controller resource type is mapped to another heat template via the resource registry in the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.yaml" target="_blank">heat environment</a>, like this:<br />
<br />
<br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"><b>resource_registry:</b><br /> OS::TripleO::SoftwareDeployment: OS::Heat::StructuredDeployment<br /> <b>OS::TripleO::Controller: puppet/controller-puppet.yaml</b><br /> OS::TripleO::Controller::Net::SoftwareConfig: net-config-bridge.yaml<br /> OS::TripleO::ControllerConfig: puppet/controller-config.yaml<br /> OS::TripleO::NodeUserData: firstboot/userdata_default.yaml<br /> </span></span><br />For clarity, I've removed the mappings not related to the controller here, and I've also not shown the resources related to configuring the cluster after initial deployment via the ResourceGroup (that will be covered in the next installment! :)<br />
<br />
I'm going to take these pieces step by step to show how the first part of the deployment flow works, starting with building one OS::TripleO::Controller.<br />
<br />
<h2>
Initial deployment flow, step by step</h2>
Creating a OS::TripleO::Controller resource creates a heat nested stack, using the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/controller-puppet.yaml" target="_blank">template defined in the resource_registry</a>.<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"></span></span><br />
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;"><br /></span></span>
The deployment flow will be familiar to anyone who has tried out Heat SoftwareConfig resources, as I covered in a <a href="http://hardysteven.blogspot.co.uk/2015/05/heat-softwareconfig-resources.html" target="_blank">previous post</a>:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdhsu41AsO-WsryOdNE4ezJBTfH2YSkt-mQmacOM6skFam1XuF2trtV9aU8vjl0MDvcW-X_T3GMcpOFaaaLl08TvzFqzHT7pcDv7St1EC35nr5XqDcT02CCv7W10RobdRbR2S5cwFhZdw/s1600/TripleOController1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="422" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdhsu41AsO-WsryOdNE4ezJBTfH2YSkt-mQmacOM6skFam1XuF2trtV9aU8vjl0MDvcW-X_T3GMcpOFaaaLl08TvzFqzHT7pcDv7St1EC35nr5XqDcT02CCv7W10RobdRbR2S5cwFhZdw/s640/TripleOController1.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAerv4E1efVVm9cqOkOoiK8suL7fA8w27V07jznT7ib82rQ0EnZPf6kAVG8mjVAkEOOJZoJv7RtRzuRSlTJYbacoqB5GeAhm0mi5H0NTWgKSrgqLp_qrZXiDNr92UHwu2moLqa7oNPpOs/s1600/TripleOController1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
<br />
The deployment sequence looks like this:<br />
<br />
<ol>
<li>An<b> OS::TripleO::NodeUserData</b> resource is created, by default this does nothing, but it provides a hook where deployers can easily plug in site specific "firstboot" configuration, e.g some special cloud-config to pass to cloud-init, or some script to run (more on this in a future post).</li>
<li>We create an <b>OS::Nova::Server</b> resource (confusingly called "Controller", the same as the ResourceGroup in the parent template..), using the flavor and size passed in to the template via parameters. Typically the "baremetal" flavor will be used, configured so the deployment happens via Ironic to enable deployment to baremetal servers.</li>
<li>An <b>OS::TripleO::SoftwareDeployment</b> is created, which applies an <b>OS::TripleO::Net::SoftwareConfig SoftwareConfig</b> resource to the server - as indicated by the names, these abstractions configure the network on the node, using the exact same method described in the <a href="http://hardysteven.blogspot.co.uk/2015/05/heat-softwareconfig-resources.html" target="_blank">primer on SoftwareConfig resources</a> - the resources are named differently to enable abstractions which cleanly support different network configurations (<a href="http://lists.openstack.org/pipermail/openstack-dev/2015-May/063938.html" target="_blank">and in future topologies</a>), e.g in the resource_registry above we'll be applying the config defined in <i>net-config-bridge.yaml</i>.</li>
<li>Last but not least, we use another <b>OS::TripleO::SoftwareDeployment</b> to apply <b>ControllerConfig</b>, which is simply passing a large map of data to <a href="https://github.com/openstack/os-apply-config" target="_blank">os-apply-config</a>, which is then stored as heiradata, (to be consumed later by puppet when it's configuring the services on the deployed cluster of nodes)</li>
</ol>
<h2>
Phew, is that all?</h2>
Well, as the eagle-eyed amongst you will have spotted, it's not - but it is the end of the initial deployment phase prior to configuring the cluster. <br />
<br />
<ul>
<li>We've deployed a node</li>
<li>Optionally performed some "firstboot" configuration</li>
<li>Configured the network on the node</li>
<li>Performed some preliminary configuration of the services on the node</li>
</ul>
This means when your ResourceGroup of OS::TripleO::Controller nodes goes to CREATE_COMPLETE state, you have a bunch of active, partially configured (but basically useless) controller nodes. <br />
<br />
The next step is to perform a series of post-deployment configuration passes on the whole ResourceGroup, or in other words configure the cluster of controllers so you have a group of fully functional OpenStack controller nodes - more on this in my next post! :)<br />
<h2>
</h2>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com1tag:blogger.com,1999:blog-7962934533489004582.post-17134376537874785402015-05-07T11:17:00.001-07:002015-05-07T11:17:53.062-07:00TripleO Heat templates Part 1 - Roles and GroupsThis is the start of a series of posts aiming to de-construct the <a href="https://github.com/openstack/tripleo-heat-templates" target="_blank">TripleO heat templates</a>, explaining the abstractions that exist,and the heat features which enable them.<br />
<br />
If you're not already a little familiar with<a href="http://hardysteven.blogspot.co.uk/2014/09/using-heat-resourcegroup-resources.html" target="_blank"> ResourceGroups</a>, "Provider Resources" used for <a href="http://docs.openstack.org/user-guide/enduser/hot-guide/hot_composition.html" target="_blank">template composition</a>, and<a href="http://hardysteven.blogspot.co.uk/2015/05/heat-softwareconfig-resources.html" target="_blank"> SoftwareConfig resources</a>, it's probably not a bad idea to check out my previous posts on those topics, as well as our <a href="http://docs.openstack.org/user-guide/enduser/hot-guide/hot.html" target="_blank">user guide</a> and <a href="http://docs.openstack.org/developer/heat/" target="_blank">other documentation</a> - TripleO makes heavy use of all of these features.<br />
<br />
<h2>
Overcloud "Roles"</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf9bzoVNbnezcxnf4KbCsO6dWl5nfZK3q3gU3PkMoXiJmEpZoHv-eRpEgj7dmG-2y3uV4cOkexriO6PxiunBYDTnlx_jiuVefkRk0Y-VYWlZsCJchRhSnxZ9RKlFqkGj1b9F8k1OO-oLE/s1600/flowRoot4911-2-0-5-9-26-3.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf9bzoVNbnezcxnf4KbCsO6dWl5nfZK3q3gU3PkMoXiJmEpZoHv-eRpEgj7dmG-2y3uV4cOkexriO6PxiunBYDTnlx_jiuVefkRk0Y-VYWlZsCJchRhSnxZ9RKlFqkGj1b9F8k1OO-oLE/s1600/flowRoot4911-2-0-5-9-26-3.png" width="640" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPpwR4IwoS9qIh_YIW3XuK2V_QNJkaIvCXgFarigm3E1zo5yeAv4sylA63lEAFqsUDbQ77Gn5jAAPDs3VTgo0dvs-b6gmmK-GsrohPEmgcKYND15kIWd6rywWhfpjCqa10pBdGhQ71HfY/s1600/TripleORoles.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPpwR4IwoS9qIh_YIW3XuK2V_QNJkaIvCXgFarigm3E1zo5yeAv4sylA63lEAFqsUDbQ77Gn5jAAPDs3VTgo0dvs-b6gmmK-GsrohPEmgcKYND15kIWd6rywWhfpjCqa10pBdGhQ71HfY/s1600/TripleORoles.png" /></a> </div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
TripleO typically refers to the deployed OpenStack cloud as the "overcloud", because the tools used to perform that deployment mirror those in the deployed cloud - e.g a small OpenStack is used to bootstrap and manage a bigger one (normally the small OpenStack is called either a "seed" or "undercloud", depending on your environment).<br />
<br />
The definition of what is deployed in your overcloud exists in a number of Heat templates, with the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml" target="_blank">top-level one</a> defining a number of groups of different node types, or "roles".<br />
<br />
<ul>
<li><b>Controller</b>: Contains API services, e.g Keystone, Neutron, Heat, Glance, Ceilometer, Horizon, and the API parts of Nova, Cinder & Swift. It can also optionally host the storage parts for Cinder, Swift and Ceph if these are not deployed separately (see below).</li>
<li><b>Compute</b>: Contains the Nova Hypervisor components</li>
<li><b>BlockStorage</b>: Contains the Cinder storage components (if not hosted on the Controller(s).</li>
<li><b>ObjectStorage</b>: Contains the Swift storage components (if not hosted on the Controllers(s).</li>
<li><b>CephStorage</b>: Contains the Ceph storage components (if not hosted on the Controllers(s).</li>
</ul>
<br />
<h2>
Roles & resource types</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9sDORH4zze__aVfhoBIRwlSjdFk7b8mDDdUbgzUdBNuM2Uiq3CZlMJH-UOA2ooKCME9Q0Mmx3tlIdTXwSE0z97PdrPogkWLXWuAWvhG_7DDne0gg-fUDf9Ou221CbOcaMxWvauaXqMO8/s1600/text12178-5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9sDORH4zze__aVfhoBIRwlSjdFk7b8mDDdUbgzUdBNuM2Uiq3CZlMJH-UOA2ooKCME9Q0Mmx3tlIdTXwSE0z97PdrPogkWLXWuAWvhG_7DDne0gg-fUDf9Ou221CbOcaMxWvauaXqMO8/s1600/text12178-5.png" width="320" /></a></div>
<br />
Each of the roles (or node types), are mapped to a a type defined in the <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.yaml#L7" target="_blank">resource_registry</a> in the environment passed to Heat.<br />
<br />
So, for example, the "Controller" role is defined in the heat template as a type <b>OS::TripleO::Controller</b>, and similar aliases exist for all the other roles.<br />
<br />
The resource registry maps this type alias to another heat template, which implements whatever is required to deploy one node with that role.<br />
<br />
So to create a node type "OS::TripleO::Controller" Heat may create a stack based on the template in "puppet/controller-puppet.yaml", or some other implementation based on whatever mapping exists in the resource_registry.<br />
<br />
This makes it very easy if you want to plug in some alternate implementation, while maintaining the top-level template interfaces and deployment topology. For example, work is currently in-progress implementing an <a href="https://review.openstack.org/#/c/178840/1/overcloud-resource-registry-docker.yaml" target="_blank">alternate implementation using docker containers</a>, as an alternative to the existing <a href="https://github.com/openstack/tripleo-heat-templates/tree/master/puppet" target="_blank">puppet</a> and <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/controller.yaml" target="_blank">element</a> based impelementations. <br />
<table class="highlight tab-size-8 js-file-line-container"><tbody>
<tr><td class="blob-code blob-code-inner js-file-line" id="LC1"><span class="pl-s"><span class="pl-ent"></span></span><br /></td><td class="blob-code blob-code-inner js-file-line" id="LC1"><span class="pl-s"><span class="pl-ent"></span></span><br /></td><td class="blob-code blob-code-inner js-file-line" id="LC2"></td><td class="blob-code blob-code-inner js-file-line" id="LC3"><br /></td><td class="blob-code blob-code-inner js-file-line" id="LC4"><br /></td><td class="blob-code blob-code-inner js-file-line" id="LC5"><br /></td><td class="blob-code blob-code-inner js-file-line" id="LC6"><br /></td><td class="blob-code blob-code-inner js-file-line" id="LC7" style="background-color: #f8eec7;"><br /></td></tr>
</tbody></table>
<h2>
Roles & ResourceGroups</h2>
<br />
Each of these roles may be independently scaled - because they are defined in an <b>OS::Heat::ResourceGroup</b>. The minimum you can deploy is one "Controller" and one "Compute" node (some roles may be deployed with zero nodes in the group).<br />
<br />
Here's an example of what that looks like in the top level <a href="https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml#L553" target="_blank">"overcloud-without-mergepy" template</a> (this is the name of the main template TripleO uses to deploy OpenStack, the "without-mergepy" part is historical and refers to an older, now deprecated, implementation.)<br />
<br />
<i><span style="font-family: "Courier New",Courier,monospace;"> Controller:<br /> type: <b>OS::Heat::ResourceGroup</b><br /> properties:<br /> count: {get_param: ControllerCount}<br /> resource_def:<br /> type: <b>OS::TripleO::Controller</b><br /> properties:<br /> AdminPassword: {get_param: AdminPassword}<br /> AdminToken: {get_param: AdminToken}</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> ...</span></i><br />
<br />
<br />
Here, you can see we've defined a group of <b>OS::TripleO::Controller</b> resources in an <b>OS::Heat::ResourceGroup</b>, and the number of nodes deployed is controlled via a template parameter, "ControllerCount", and similarly a number of template parameters are referenced to provide input properties to enable configuration of the deployed controller node (I've abbreviated the full list of properties).<br />
<br />
This pattern is repeated for all roles, so building a specified number of nodes for a particular role (or adding/removing them via a stack-update), is as simple as passing a different number into Heat as a stack parameter :)<br />
<br />
<h2>
That's all, folks</h2>
<br />
That's all for today - hopefully it provides an overview of the top-level interfaces provided by the TripleO Heat templates, and illustrates the following:<br />
<br />
<ul>
<li>There are clearly defined node "roles", containing the various parts of your OpenStack deployment</li>
<li>The patterns used to define and implement these roles are repeated, which helps understand the templates despite them being fairly large.</li>
<li>The implementation is modular, and abstractions exist which make implementing different "back end" implementations relatively simple.</li>
<li>Deployments can be easily scaled due to using Heat's ResourceGroup functionality.</li>
</ul>
In future instalments I'll dig further into the individual node implementations, ways to easily plug in site-specific additional configuration, and ways in which you can control and validate the deployments performed via TripleO.Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com231tag:blogger.com,1999:blog-7962934533489004582.post-40587985311138291152015-05-05T09:15:00.000-07:002015-05-05T09:15:30.456-07:00Heat SoftwareConfig resources - primer/overview.In this post, I'm going to provide an overview of Heat's Software Configuration resources, as a preface to digging in more detail into the structure of <a href="https://github.com/openstack/tripleo-heat-templates" target="_blank">TripleO heat templates</a>, which leverage SoftwareConfig functionality to install and configure the deployed OpenStack cloud.<br />
<br />
Heat has supported <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareConfig" target="_blank">SoftwareConfig</a> and <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployment" target="_blank">SoftwareDeployment</a> resources since the <a href="https://wiki.openstack.org/wiki/ReleaseNotes/Icehouse#Heat" target="_blank">Icehouse</a> release, in an effort to provide flexible and non-opinionated abstractions which enable integration with existing software configuration tools and scripts.<br />
<br />
<a name='more'></a><br />
The key concepts and some examples are <a href="http://docs.openstack.org/user-guide/enduser/hot-guide/hot_software_deployment.html" target="_blank">described in our user guide</a>, but what follows is more of a primer, to provide necessary context before we get in to decomposing the TripleO heat templates. If what follows looks a little scary, check out my <a href="https://www.youtube.com/watch?v=PChya-fe0iw" target="_blank">introductory (very simple) screencast</a>, which was recorded for the <a href="https://www.youtube.com/watch?v=fw0JhywwA1E" target="_blank">Heat Beyond the Basics</a> session at the Paris OpenStack summit.<br />
<br />
<h2>
Heat SoftwareConfig resources</h2>
<br />
There are three resources necessary in a typical software configuration scenario:<br />
<br />
<ol>
<li>An <b>OS::Heat::SoftwareConfig</b> resource - this encapsulates the config to be applied, e.g a script, puppet manifest, or any other config definition format you care to use. This is just a wrapper for the config to apply, optionally parameterized with input values, it doesn't actually configure anything.</li>
<li>An <b>OS::Heat::SoftwareDeployment</b> resource - this is the thing which actually applies the config from (1) - when it moves to and IN_PROGRESS state, it makes the config available to the specified server. By default, the deployment will stay in the IN_PROGRESS state until a signal is received via the Heat API, notifying the service of success (or failure..) applying the config.</li>
<li>An <b>OS::Nova::Server</b> resource - this is the instance (or physical server in the case of TripleO deploying via Nova and Ironic) being configured, it must contain some tools to support SoftwareConfig, as discussed below, and define the <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server-props" target="_blank">user_data_format property</a> to enable SoftwareConfig.</li>
</ol>
<br />
There are also <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::StructuredConfig" target="_blank">OS::Heat::StructuredConfig</a> and <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::StructuredDeployment" target="_blank">OS::Heat::StructuredDeployment</a> resources, which are basically identical to SoftwareConfig/SoftwareDeployment resources, except the config is defined as a map, rather than a string (which is useful for some tools such as os-apply-config, which consumes a map of config data, rather than a string such as is used e.g applying a script or a puppet manifest). So I'll refer only to SoftwareConfig/SoftwareDeployment from here on, but all the concepts are applicable to both.<br />
<br />
<h2>
SoftwareDeployment flow </h2>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7CDQapGarRZH7LBLWWIZyGFnZ273B66ArZi9CMyahY13hyh4ubNO7KymBrhRbfqiAiwHOnKuXrDjizkQ3dAJ4Via51bKJJ6Ib9Q_yredBVJ5kQCQ-W4tEtx2xoEi15CL0cSV_4Bs1I7o/s1600/SoftwareConfigFlow2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7CDQapGarRZH7LBLWWIZyGFnZ273B66ArZi9CMyahY13hyh4ubNO7KymBrhRbfqiAiwHOnKuXrDjizkQ3dAJ4Via51bKJJ6Ib9Q_yredBVJ5kQCQ-W4tEtx2xoEi15CL0cSV_4Bs1I7o/s640/SoftwareConfigFlow2.png" height="232" width="640" /></a></div>
<br />
<br />
<br />
The server being configured requires some agents in order to collect and process the configuration made available via the deployment. Typically, this works as follows:<br />
<br />
<ol>
<li>os-collect-config polls the Heat API for updated resource metadata associated with the OS::Nova::Server resource</li>
<li>When metadata is updated, os-refresh-config runs, and triggers an element called <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/elements/heat-config" target="_blank">heat-config</a>.</li>
<li>heat-config then uses the "group" property defined in the <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareConfig-props" target="_blank">SoftwareConfig properties</a> to process applying the configuration via a <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/elements" target="_blank">hook script</a>. Heat provides (via the <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/elements" target="_blank">heat-templates</a> repo) a variety of ready-made hook scripts for some popular tools, but it's simple to write your own if needed, and it involves no changes to Heat, only a script inside the image you deploy.</li>
<li>On completion, the hook uses "heat-config-notify" to send a signal back to Heat, which includes the return code of the thing we ran, along with stdout/stderr from the tool invoked by the hook script. This information is then made available via "heat deployment-show" via the Heat API.</li>
<li>If the signal notified success, we move the SoftwareDeployment to COMPLETE state, e.g CREATE_COMPLETE. If it failed, we move it to e.g CREATE_FAILED state.</li>
</ol>
<h2>
SoftwareDeployment HOT template definition</h2>
We have <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates" target="_blank">example templates</a> for many popular tools, but one thing to emphasize is that the template definition is not dependent on the tool doing the configuration - the coupling between the config to apply and the tool applying it only happens inside the instance via the heat-config hook logic.<br />
<br />
Looking at the <a href="https://github.com/openstack/heat-templates/blob/master/hot/software-config/example-templates/example-puppet-template.yaml" target="_blank">puppet example</a> step-by step:<br />
<br />
<h3>
1. Define the SoftwareConfig resource</h3>
<span style="font-family: "Courier New",Courier,monospace;"><i> config:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> type: <b>OS::Heat::SoftwareConfig</b></i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> properties:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> group: puppet</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> inputs:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> - name: foo</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> default: aninput</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> - name: bar</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> outputs:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> - name: result</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> config:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> get_file: config-scripts/example-puppet-manifest.pp</i></span><br />
<br />
Here, we can see the following:<br />
<ul>
<li>We define the SoftwareConfig, which references a puppet manifest via get_file. </li>
<li>We parameterize applying the manifest by providing some input values, which can specify a default value</li>
<li>The "group" is specified as "puppet", which will enable <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/elements/heat-config" target="_blank">heat-config</a> to correctly apply the manifest using the <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/elements/heat-config-puppet" target="_blank">heat-config-puppet hook</a>.</li>
<li>We specify an output - this means inside<a href="https://github.com/openstack/heat-templates/blob/master/hot/software-config/example-templates/config-scripts/example-puppet-manifest.pp" target="_blank"> the manifest </a>we can reference the special "heat_outputs_path" input, and write a file containing a result related to this output.</li>
</ul>
A SoftwareConfig, when created, goes to CREATE_COMPLETE immediately, there is no dependency on applying the config to a server, thus a config once defined may be applied to multiple servers (potentially with different input parameters).<br />
<br />
<h3>
2. Define the Server resource</h3>
<span style="font-family: "Courier New",Courier,monospace;"><i> server:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> type: <b>OS::Nova::Server</b></i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> properties:</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> image:animage</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> flavor:m1.small</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><i> user_data_format: SOFTWARE_CONFIG</i></span><br />
<br />
Here, the some things to note:<br />
<ul>
<li>The image provided must contain the tools previously discussed (os-collect-config, os-refresh-config, heat-config and whatever hooks you need for the "group" properties you want to specify)</li>
<li>user_data_format must be set to "SOFTWARE_CONFIG"</li>
<li>user_data may still be specified - this is useful where you want to combine first-boot configuration (e.g via cloud-init) with subsequent application deployment via SoftwareConfig. A <a href="https://review.openstack.org/#/c/170137/" target="_blank">recent TripleO patch</a> illustrates this approach. </li>
</ul>
Optionally you can also specify the transport used for polling metadata via <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server-props" target="_blank">software_config_transport</a> - by default heat will poll via the heat-api-cfn API, but you can also poll via the native heat-api, or in Kilo Heat, via Swift. <br />
<h4>
</h4>
<h3>
3. Define the SoftwareDeployment resource</h3>
<i><span style="font-family: "Courier New",Courier,monospace;"> deployment:</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> type: <b>OS::Heat::SoftwareDeployment</b></span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> properties:</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> config:</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> get_resource: config</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> server:</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> get_resource: server</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> input_values:</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> foo:abc</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> bar:xyz</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> actions:</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"> - CREATE</span></i><br />
<i><span style="font-family: "Courier New",Courier,monospace;"><br /></span></i>
<br />
<h4>
</h4>
I've put the SoftwareDeployment resource last on purpose - to highlight the normal deployment flow:<br />
<ul>
<li>heat creates the SoftwareConfig, then the Server, then the Deployment</li>
<li>The Deployment applies the config, depending on both the SoftwareConfig and Server resources (implicitly via the get_resource references).</li>
<li>The Deployment resource status depends on the status of the signal sent back to heat after applying the config (via the heat-config hook).</li>
</ul>
Note the "actions" property is optional (defaults to applying the config on both CREATE and UPDATE actions, but it can be used to specify that, for example you only want configuration to happen on create and not update (as above), or perhaps that you only want to run some cleanup on DELETE, for example unregistering the server from some external service.<br />
<br />
input_values is used to provide the values for the parameters defined in the SoftwareConfig resource - thus it's perfectly fine for multiple Deployment resources to reference the same Config resource (potentially with different input_values and/or actions specified), but each deployment references exactly one Config (e.g a SoftwareDeployment cannot apply multiple SoftwareConfig resources, only one).<br />
<br />
<h2>
Dealing with dependencies</h2>
<h2>
</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUAk8Pj4eLnNUcnjNOzeM1fWj0JHlqtnTqo4VXXc4LK3cQBbg174hHHbfV2q1mK6fHBIjyElf_15BgwOx2Mb1qgSx-BQ9vw5t_O4sPYxgJJg7g00wTk3GPUNSfqXCx5iXOS3etoCp2kEk/s1600/SoftwareConfigFlow3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUAk8Pj4eLnNUcnjNOzeM1fWj0JHlqtnTqo4VXXc4LK3cQBbg174hHHbfV2q1mK6fHBIjyElf_15BgwOx2Mb1qgSx-BQ9vw5t_O4sPYxgJJg7g00wTk3GPUNSfqXCx5iXOS3etoCp2kEk/s1600/SoftwareConfigFlow3.png" height="266" width="640" /></a></div>
<h3>
<span id="goog_2010432556"></span><span id="goog_2010432557"></span></h3>
A common pattern is doing a series of configuration steps, for example configure a database, then some application that uses the database (and requires it to be installed and configured).<br />
<br />
There are a couple of ways to handle this:<br />
<br />
<ol>
<li>SoftwareDeployment resources have a "name" property, which can influence the sort-order so that, for example, heat-config will apply "config1" before "config2".</li>
<li>Template directive "depends_on" can be used to specify an explicit dependency between two (or more) SoftwareDeployment resources (<b>Note: </b>the dependency is between the *deployment* resources, not the SoftwareConfig resources!)</li>
</ol>
My preference is to use "depends_on" in most cases - it provides the most explicit control of deployment serialization, and makes the ordering very clear from the template level.<br />
<br />
<h2>
Conclusion and further resources</h2>
Hopefully that concludes a reasonable overview of heat SoftwareConfiguration capabilities. For further information, please see the heat documentation, the <a href="http://docs.openstack.org/user-guide/enduser/hot-guide/hot_software_deployment.html" target="_blank">user guide</a> and <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployment" target="_blank">template guide</a> and <a href="https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates" target="_blank">example templates</a> are a good starting point.<br />
<br />
If you're feeling brave, you can also dive into the <a href="https://github.com/openstack/tripleo-heat-templates" target="_blank">TripleO heat templates</a>, which make extensive use of SoftwareConfig/StructuredConfig and SoftwareDeployment/Structured deployment resources. More on TripleO specifically in a future post! :)<br />
<h4>
</h4>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com4tag:blogger.com,1999:blog-7962934533489004582.post-84094657044703932212015-04-20T09:16:00.000-07:002015-05-18T08:19:18.526-07:00Debugging TripleO Heat templatesLately, I've been spending increasing amounts of time working with <a href="https://github.com/openstack/tripleo-heat-templates" target="_blank">TripleO heat templates</a>, and have noticed some recurring aspects of my workflow whilst debugging them which I thought may be worth sharing.<br />
<br />
For the uninitiated, <a href="https://wiki.openstack.org/wiki/TripleO" target="_blank">TripleO</a> is an OpenStack deployment project, which aims to deploy and manage <a href="https://www.openstack.org/" target="_blank">OpenStack</a> using standard <a href="http://developer.openstack.org/api-ref.html" target="_blank">OpenStack API's</a>. In practice, this means using <a href="https://wiki.openstack.org/wiki/Nova" target="_blank">Nova</a> and <a href="https://wiki.openstack.org/wiki/Ironic" target="_blank">Ironic</a> for baremetal node provisioning, and <a href="https://wiki.openstack.org/wiki/Heat" target="_blank">Heat</a> to orchestrate the deployment and configuration of the nodes.<br />
<br />
The TripleO heat templates, unlike most of the heat <a href="https://github.com/openstack/heat-templates" target="_blank">examples</a>, are pretty complex. They make extensive use of many "advanced" features, such as nested stacks, using <a href="http://hardysteven.blogspot.co.uk/2013_10_01_archive.html" target="_blank">provider resources</a> via the <a href="http://docs.openstack.org/developer/heat/template_guide/environment.html" target="_blank">environment</a> and also many <a href="http://docs.openstack.org/hot-guide/content/software-deployment-resources.html" target="_blank">software config</a> resources.<br />
<br />
This makes TripleO a fairly daunting target to those wishing to debug and modify and/or debug the TripleO templates.<br />
<br />
Fortunately TripleO templates, although large, have many repeated patterns, and good levels of abstraction and modularity. Combined with some recently added heat interfaces, it becomes rapidly less daunting, as I will demonstrate in the worked example below:<br />
<br />
<br />
<a name='more'></a><br />
<h2>
</h2>
<h3>
<span style="font-family: inherit;">Step 1: Create the Stack </span></h3>
<br />
So, step 1 when deploying OpenStack via TripleO is to do a "heat stack-create". Whether you create the heat stack directly via <a href="https://github.com/openstack/python-heatclient" target="_blank">python-heatclient</a> (which is what the TripleO "<a href="https://github.com/openstack/tripleo-incubator/blob/master/scripts/devtest_overcloud.sh" target="_blank">devtest</a>" script calls), or indirectly via some other interface such as <a href="https://github.com/openstack/tuskar-ui" target="_blank">tuskar-ui</a> the end result is the same - a heat stack is created (normally it's called "overcloud" by default):<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat stack-create -e /home/shardy/tripleo/overcloud-env.json -e /home/shardy/tripleo/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -t 360 -f /home/shardy/tripleo/tripleo-heat-templates/overcloud-without-</span><br />
<span style="font-family: "Courier New",Courier,monospace;">mergepy.yaml -P ExtraConfig= overcloud</span><br />
<span style="font-family: "Courier New",Courier,monospace;"></span><br />
<br />
<br />
+--------------------------------------+------------+--------------------+----------------------+<br />
| id | stack_name | stack_status | creation_time |<br />
+--------------------------------------+------------+--------------------+----------------------+<br />
| e4cfc4a8-d9e9-4033-8556-5ebca84c1455 | overcloud | CREATE_IN_PROGRESS | 2015-04-20T11:05:53Z |<br />
+--------------------------------------+------------+--------------------+----------------------+<br />
<h2>
<span style="font-family: inherit;"> </span></h2>
<h3>
<span style="font-family: inherit;">Step 2: Oh No - CREATE_FAILED! </span></h3>
Ok, it happens - sometimes you have a fault in your environment, a bug in your templates, or just get bitten by a regression in one of the projects used to deploy your overcloud.<br />
<br />
Unfortunately that modularity I just mentioned in the templates leads to a level of additional complexity when debugging - the tree of resources created by heat is actually grouped into nearly 40 nested stacks! (In my environment, this number is dependent on the number of nodes you're deploying).<br />
<br />
You can see them all, including which one failed, with heat stack-list, using the --show-nested option, and your choice of either grep "FAILED" or the -f filter option to python heatclient:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat stack-list --show-nested -f "status=FAILED"</span><br />
<span style="font-family: "Courier New",Courier,monospace;">+--------------------------------------+----------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+<br />| id | stack_name | stack_status | creation_time | parent |<br />+--------------------------------------+----------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+<br />| e4cfc4a8-d9e9-4033-8556-5ebca84c1455 | <b>overcloud </b> | CREATE_FAILED | 2015-04-20T11:05:53Z | None |<br />| 36f3ef93-872f-460b-bd6a-14a89569d5a7 | overcloud-<b>ControllerNodesPostDeployment</b>-rl67kiqu7pbp | CREATE_FAILED | 2015-04-20T11:09:18Z | e4cfc4a8-d9e9-4033-8556-5ebca84c1455 |<br />| 28d1fd38-85ba-442b-9e57-859731349e94 | overcloud-ControllerNodesPostDeployment-rl67kiqu7pbp-<b>ControllerDeploymentLoadBalancer_Step1</b>-tnsuslbx5hu7 | CREATE_FAILED | 2015-04-20T11:09:20Z | 36f3ef93-872f-460b-bd6a-14a89569d5a7 |<br />+--------------------------------------+----------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------------------------------+</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: inherit;">Here, we can derive some useful information looking at the stack names, note that in all cases we can disregard the randomly generated suffix on the stack names (heat adds it internally for nested stack resources). </span><br />
<span style="font-family: inherit;"></span><br />
<ul>
<li><span style="font-family: inherit;"><b>overcloud</b> is the top-level stack, the parent at the top of the tree. This is defined by the <a href="https://www.blogger.com/goog_1492443106">overcloud-without-</a></span><span style="font-family: inherit;"><a href="https://github.com/hardys/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml" target="_blank">mergepy.yaml </a></span><br />
<span style="font-family: inherit;">template which we passed to heat stack-create.</span></li>
<li><span style="font-family: inherit;"><b>ControllerNodesPostDeployment-rl67kiqu7pbp</b> is the nested stack which handles post-deployment configuration of all Controller nodes. This is the <b>ControllerNodesPostDeployment resource</b>, defined by the <a href="https://github.com/hardys/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.yaml#L9" target="_blank">overcloud resource registry</a> as the implementation of the <i>OS::TripleO::ControllerPostDeployment </i>type, which is a provider resource alias for <a href="https://github.com/hardys/tripleo-heat-templates/blob/master/puppet/controller-post-puppet.yaml" target="_blank">this template</a> when using the puppet implementation.</span></li>
<li><span style="font-family: inherit;">The final (verbosely named!) stack maps to the <br /><b>ControllerDeploymentLoadBalancer_Step1 resource</b> in controller-post-puppet.yaml.</span></li>
</ul>
<span style="font-family: inherit;">All of this is a long-winded way of saying that something went wrong applying a puppet manifest, via an OS::Heat::StructuredDeployments resource (<br /><b>ControllerDeploymentLoadBalancer_Step1</b>) - anything with "Deployment" in the name failing is highly likely to mean the same thing.</span><br />
<br />
<span style="font-family: inherit;">Armed with this information, we can proceed to figure out why :) </span><br />
<span style="font-family: inherit;"></span><br />
<h2>
<span style="font-family: inherit;"> </span></h2>
<h3>
<span style="font-family: inherit;">Step 3: Resource Introspection</span></h3>
<span style="font-family: inherit;">So we now know which nested stack failed, but not which resource, or why.</span><br />
<br />
<span style="font-family: inherit;">There's a couple of ways to find this out, you can either use the steps outlined in my <a href="http://hardysteven.blogspot.co.uk/2013/08/heat-nested-resource-introspection.html" target="_blank">previous post</a> about nested resource introspection, or (if you're lazy like me), you can use the heat resource-list --nested-depth option to save some time:</span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;">$ heat resource-list --nested-depth 5 overcloud | grep FAILED<br />| ControllerNodesPostDeployment | 36f3ef93-872f-460b-bd6a-14a89569d5a7 | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2015-04-20T11:05:53Z | |<br />| <b>ControllerDeploymentLoadBalancer_Step1</b> | 28d1fd38-85ba-442b-9e57-859731349e94 | <b>OS::Heat::StructuredDeployments</b> | CREATE_FAILED | 2015-04-20T11:09:19Z | <b>ControllerNodesPostDeployment</b> |<br />| <b>0</b> | 980137bc-21b1-460c-9d4a-488cb5611a6c | <b>OS::Heat::StructuredDeployment</b> | CREATE_FAILED | 2015-04-20T11:09:20Z | <b>ControllerDeploymentLoadBalancer_Step1</b> |</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"><br /></span></span>
<span style="font-family: inherit;">Here, we can see several things:</span><br />
<ul>
<li><span style="font-family: inherit;"><b>ControllerDeploymentLoadBalancer_Step1 </b>has failed, it's an <br /><i>OS::Heat::StructuredDeployments </i>resource. StructuredDeployments (plural) resources apply a heat StructuredConfig/SoftwareConfig to a group of servers.</span></li>
<li><span style="font-family: inherit;">There's a <b>"0" </b>resource, which is a <br /><i>OS::Heat::StructuredDeployment</i><b> </b>(singular) type. The parent resource (last column) of this is <b>ControllerDeploymentLoadBalancer_Step1.</b> This is because a SoftwareDeployments resource creates a nested stack with a (sequentially named) SoftwareDeployment per server (in this case, one per Controller node in the <i>OS::Heat::ResourceGroup </i>defined as "Controller" in the <a href="https://github.com/hardys/tripleo-heat-templates/blob/master/overcloud-without-mergepy.yaml#L515" target="_blank">overcloud-without-mergepy template</a>)</span></li>
</ul>
<span style="font-family: inherit;"></span><br />
<span style="font-family: inherit;">Now, we can do a resource-show to find out the reason for the failure. Here, we use the ID of <br />ControllerDeploymentLoadBalancer_Step1 as the stack ID, because all nested stack resources set the ID as that of the stack they create:</span><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ heat resource-show 28d1fd38-85ba-442b-9e57-859731349e94 0 | grep resource_status_reason<br />| resource_status_reason | Error: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 </span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"><br /></span></span>
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: inherit;">So, to summarize what we've discovered so far</span><br />
<span style="font-family: inherit;"></span><br />
<ul>
<li><span style="font-family: inherit;">A SoftwareDeployment (in this case a puppet run) failed on Controller node 0</span></li>
<li><span style="font-family: inherit;">The thing it was running exited with status code 6. </span></li>
</ul>
<span style="font-family: inherit;">The next step is to look at the logs to work out why..</span><span style="font-family: inherit;"></span><br />
<span style="font-family: inherit;"></span><br />
<h2>
<span style="font-family: inherit;"> </span></h2>
<h3>
<span style="font-family: inherit;">Step 4: Debugging the failure</span></h3>
<span style="font-family: inherit;">When a Heat SoftwareDeployment resource is triggered, it runs something on the node (e.g applying a puppet manifest), then signals either success or failure back to Heat. Fortunately, in recent version of Heat, there is an API which exposes this information (in a more verbose way than the resource-show output above with the reason for failure):</span><br />
<br />
<span style="font-family: inherit;">To access it, you need the ID of the deployment (e.g <br />980137bc-21b1-460c-9d4a-488cb5611a6c) from the heat resource-list above):</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"><br /></span></span>
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;"></span></span><br />
<span style="font-family: "Courier New",Courier,monospace;">heat deployment-show 980137bc-21b1-460c-9d4a-488cb5611a6c<br />{<br /> "status": "FAILED",<br /> "server_id": "6a025200-b20e-47df-ae4c-97a54499b586",<br /> "config_id": "b924d133-42d7-48ab-b2c9-7311de3b3ca4",<br /> "output_values": { <br /> "deploy_stdout": "<stdout of command>,<br /> "deploy_stderr": "<stderr of command>",<br /> "deploy_status_code": 6<br /> },<br /> "creation_time": "2015-04-20T11:09:20Z",<br /> "updated_time": "2015-04-20T11:10:02Z",<br /> "input_values": {},<br /> "action": "CREATE",<br /> "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 6",<br /> "id": "980137bc-21b1-460c-9d4a-488cb5611a6c"<br />}</span><br />
<span style="font-family: inherit;">I've not included the full stderr/stdout because it's pretty long, but it's basically the same information that you get from SSHing onto the node and looking at the logs.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><span style="font-family: inherit;">If you still want to do that, you can use "nova show" with the "server_id" above to get the IP of the node, SSH in and do further investigations.</span></span><br />
<br />
<h3>
<span style="font-family: inherit;">In Summary...</span></h3>
<span style="font-family: inherit;">So those paying attention will have spotted that this all really boils down to two steps:</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<ol>
<li><span style="font-family: inherit;"> Use heat resource-list with the --nested-depth option to get the failing resource. The one you want is the one which isn't the parent_resource to any other and is in a FAILED state.</span></li>
<li><span style="font-family: inherit;">Investigate what caused the failure, for failing SoftwareDeployment resources heat deployment-show is a useful time-saver which avoids always needing to log on to the node.</span></li>
</ol>
<span style="font-family: inherit;"><br />Hopefully this somewhat demystifies the debugging of TripleO templates, and other large Heat deployments which use similar techniques such as nested stacks and SoftwareConfig resources!</span><span style="font-family: "Courier New",Courier,monospace;"></span><br />
<br />
<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"></span><span style="font-family: "Courier New",Courier,monospace;"></span>Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com117tag:blogger.com,1999:blog-7962934533489004582.post-38955410481646434322014-09-11T03:48:00.001-07:002014-09-11T03:48:40.508-07:00Using Heat ResourceGroup resources<div>
<br /></div>
This has come up a few times recently, so I wanted to share a howto showing how (and how not) to use the group resources in Heat, e.g OS::Heat::ResourceGroup and OS::Heat::AutoScalingGroup.<br />
<br />
The key thing to remember when dealing with these resources is that they can multiply any number of resources (expressed as a heat stack), not just individual resources. This is a very cool feature when you get your head around it! :)<br />
<br />
Lets go through a worked example, where we use ResourceGroup to create 5 identical servers, each with a cinder volume of the same size attached.<br />
<br />
<a name='more'></a><br />
<h3>
Resource group basics</h3>
<br />
To create one server with a volume attached, you define the server, a volume, and a volume attachment resource, like this:<br />
<br />
<script src="https://gist.github.com/hardys/51bfc48349340fe1a857.js"></script>
<br />
<div>
Now, lets say you need 5 (or 500) of these identical servers with an attached volume. What you do *not* want to do is create three groups of resources (Server, Volume and VolumeAttachment), and somehow try to connect them all together. This is an anti-pattern which will cause you much pain and frustration! :)<br />
<br />
Instead, you need to use ResourceGroup to scale out the combination of resources. Fortunately, Heat makes this very easy to do. Lets say you call the template above creating one server with attached volume <i>server_with_volume.yaml</i>, you can create 5 identical nested stacks, each containing one server, volume and volume attachment like this:<br />
<br />
<script src="https://gist.github.com/hardys/83307cee5eace5fb1a57.js"></script>
<i><b>Note:</b> currently templates referencing nested stack templates can only be launched via python-heatclient (not the Horizon dashboard, a known issue we're working on resolving).</i><br />
<br />
Simply do <i>heat stack-create my_group -f server_with_volume_group.yaml</i> and Heat will create 5 identical servers, attached to 5 identical volumes!<br />
<br />
A more complete example related to the fragments above is available <a href="https://github.com/hardys/demo_templates/tree/master/juno_summit_intro_to_heat/example3_server_with_volume_group" target="_blank">here</a>.</div>
<h3>
<br />
Resource groups and provider resources</h3>
<br />
What's that you say? You don't like the nested stack reference hard-coded template name? No problem! :)
You can also make use of the <a href="http://docs.openstack.org/developer/heat/template_guide/environment.html" target="_blank">environment</a> to define a <a href="http://hardysteven.blogspot.co.uk/2013_10_01_archive.html" target="_blank">provider resource type alias</a>.<br />
<br />
<script src="https://gist.github.com/hardys/d9f1a3edf01ebbf9107b.js"></script>
Then specify the type alias instead of the template name in the ResourceGroup definition:
<br />
<script src="https://gist.github.com/hardys/0baa0fc8cdd5455951cf.js"></script>
<br />
This can be lauched like this<i>heat stack-create my_group2 -f server_with_volume_group.yaml -e env_server_with_volume.yaml</i><br />
<br />
The example will work exactly as before, only different versions of My::Server::WithVolume can easily be substituted, for example if you need a staging workflow where the resource alias is reused across a large number of templates, different versions of the nested template can easily be specified by changing it in one place (the environment).<br />
<br />
That is all, for more information, please see the examples in the <a href="https://github.com/openstack/heat-templates" target="_blank">heat-templates</a>, and this <a href="https://review.openstack.org/#/c/119015/" target="_blank">new example</a> which shows how to attach several identical volumes to one server.Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com36tag:blogger.com,1999:blog-7962934533489004582.post-3041874216178513092014-04-16T03:14:00.000-07:002014-04-16T03:14:03.524-07:00Heat auth model updates - part 2 "Stack Domain Users"As promised, here's the second part of my updates on the Heat auth model, following on from <a href="http://hardysteven.blogspot.co.uk/2014/04/heat-auth-model-updates-part-1-trusts.html" target="_blank">part 1 describing our use of Keystone trusts.</a><br />
<br />
This post will cover details of the recently implemented <a href="https://blueprints.launchpad.net/heat/+spec/instance-users" target="_blank">instance-users blueprint</a>, which makes use of <a href="https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3.md#domains-v3domains" target="_blank">keystone domains</a> to contain users related to credentials which are deployed inside instances created by heat. If you just want to know how the new stuff works, you can skip to the last sections :)<br />
<br />
<h3>
So...why does heat create users at all?</h3>
<div>
Lets start with a bit of context. Heat has historically needed to do some or all of the following:</div>
<div>
<ol>
<li>Provide metadata to agents inside instances, which poll for changes and apply the configuration expressed in the metadata to the instance.</li>
<li>Signal completion of some action, typically configuration of software on a VM after it is booted (because nova moves the state of a VM to "Active" as soon as it spawns it, not when heat has fully configured it)</li>
<li>Provide application level status or metrics from inside the instance, e.g to allow AutoScaling actions to be performed in response to some measure of performance or quality of service.</li>
</ol>
<div>
Heat provides API's which enable all of these things, but all of those API's require some sort of authentication, e.g credentials so whatever agent is running on the instance is able to access it. So credentials must be deployed inside the instance, e.g here's how things work if you're using the heat-cfntools agents:</div>
</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGuzxO77B2M8sCkHhOSx9XCtzWYKsvkUtf6xxkLUgXgMb2yRZhchFAhEcXYMWHmoK8cYZsiLLP10h8jOCvxCU0EN6ZSJecvyiqXnFpJX5wUik-ZpU8BlkUjozY013VBCntOewgiLAnEco/s1600/software_config5_small.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGuzxO77B2M8sCkHhOSx9XCtzWYKsvkUtf6xxkLUgXgMb2yRZhchFAhEcXYMWHmoK8cYZsiLLP10h8jOCvxCU0EN6ZSJecvyiqXnFpJX5wUik-ZpU8BlkUjozY013VBCntOewgiLAnEco/s1600/software_config5_small.png" height="298" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">heat-cfntools agents data-flow with CFN-compatible API's</td></tr>
</tbody></table>
<div>
<br />
<a name='more'></a><br /></div>
<div>
The heat-cfntools agents use signed requests, which requires an ec2 keypair created via keystone, which is then used to sign requests to the heat cloudformation and cloudwatch compatible API's, which are authenticated by heat via signature validation (which uses the keystone ec2tokens extension).</div>
<div>
<br /></div>
<div>
The problem is, ec2 keypairs are associated with a user. And we don't want to deploy credentials directly related to the stack owner, otherwise any compromise of the (implicitly untrusted) instance could result in a cascading compromise where an attacker could take control of anything the stack-owning user has permission to access.</div>
<div>
<br /></div>
<div>
I've used cfntools/ec2tokens as an example, but the same issue exists if you use any credential available via keystone (token, username/password) which can be used to authenticate with the heat APIs.</div>
<div>
<br /></div>
<div>
So we need separation/isolation of the credentials deployed in the instance, such that we can limit the access allowed to the minimum necessary to make heat work. Our first attempt at this did the following:</div>
<div>
<ul>
<li>Create a new user in keystone, in the same project as the stack owner (either explicitly in the template via <a href="http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::IAM::User" target="_blank">User</a> and <a href="http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::IAM::AccessKey" target="_blank">AccessKey</a> resources, or for some resources such as<a href="http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::CloudFormation::WaitConditionHandle" target="_blank"> WaitConditionHandle</a> and <a href="http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::ScalingPolicy" target="_blank">ScalingPolicy</a> we do it internally to obtain an ec2 keypair for generation of a pre-signed URL)</li>
<li>Add the "heat stack user" to a special role, default "heat_stack_user" (configurable via the heat_stack_user_role in heat.conf)</li>
<li>Limit the API surface accessible to the "heat_stack_user" <a href="https://github.com/openstack/heat/blob/master/etc/heat/policy.json#L3" target="_blank">via policy.json</a>, with the expectation that access to other service's will be restricted in a similar way, or denied completely via network separation/firewall rules.</li>
</ul>
<div>
This approach is flawed, and led to this <a href="https://bugs.launchpad.net/heat/+bug/1089261" target="_blank">long-standing bug</a>, there are multiple problems:</div>
</div>
<div>
<ul>
<li>It requires the user creating the stack to have permissions to create users in keystone, which typically requires administrative roles.</li>
<li>It doesn't provide complete separation - even with the policy rules, it's possible a compromised stack could abuse the credentials (for example obtaining metadata for some other stack created by the user in the same project)</li>
<li>It clutters the user list for the project with spurious (from the user/operator perspective) users who aren't "real" users, the users are a heat implementation detail, and we're exposing them to the end user.</li>
</ul>
</div>
<div>
<br /></div>
<h3>
Hmm, that sounds bad, what's the alternative?</h3>
<div>
Well, we've been considering that for quite some time ;) <a href="https://wiki.openstack.org/wiki/Heat/Blueprints/InstanceUsers" target="_blank">multiple solutions were discussed</a>:</div>
<div>
<ul>
<li>Delegating a subset of user roles via trusts (rejected because token expiry is not optional, and separation from the stack owner is desired, e.g we don't really want to delegate or impersonate them from the instance, we just need an identity which can be verified as related to the stack)</li>
<li>Rolling our own auth mechanism based on some random "token" (some folks were in favour of this, but I'm opposed to it, I think we should stick to orchestration and leverage or improve what's in keystone instead of taking on the burden and security risk of maintaining our own auth scheme)</li>
<li>Using the keystone OAuth extension to use OAuth keypairs and signed requests. (This was rejected due to lack of keystoneclient support, e.g client API and auth middleware, maybe we'll revisit enabling this as an option in some future release).</li>
<li>Isolating the in-instance users by creating them in a completely separate heat-specific keystone domain. This idea was first suggested by <a href="http://adam.younglogic.com/category/software/openstack/" target="_blank">Adam Young</a>, as is what we ended up <a href="https://blueprints.launchpad.net/heat/+spec/instance-users" target="_blank">implementing for Icehouse.</a></li>
</ul>
<h2>
</h2>
<h3>
"Stack Domain Users", the details..</h3>
</div>
<div>
The new approach is, effectively, an optimisation of the existing implementation. We encapuslate all stack-defined users (ie users created as a result of things contained in a heat template) in a separate domain, which is created specifically to contain things related only to heat stacks. A user is created which is the "domain admin", and heat uses that user to manage the lifecycle of the users in the "stack user domain".</div>
<div>
<br /></div>
<div>
There are two aspects of this I'll discuss below, firstly what deployers need to do to enable stack domain users in Heat (Icehouse or later), and secondly what actually happens when you create a stack, and how it addresses the previously identified problems:<br />
<br /></div>
<h3>
</h3>
<h4>
When deploying heat:</h4>
<div>
<ul>
<li>A special keystone domain is created, e.g one called "heat" and the ID is set in the "stack_user_domain" option in heat.conf</li>
<li>A user with sufficient permissions to create/delete projects and users in the "heat" domain is created, e.g <a href="https://github.com/openstack-dev/devstack/blob/master/lib/heat#L279" target="_blank">in devstack</a> a user called "heat_domain_admin" is created, and given the admin role on the heat domain.</li>
<li>The username/password for the domain admin user is set in heat.conf (stack_domain_admin and stack_domain_admin_password). This user administers "stack domain users" on behalf of stack owners, so they no longer need to be admins themselves, and the risk of this escalation path is limited because the heat_domain_admin is only given administrative permission for the "heat" domain.</li>
</ul>
<div>
This is all done automatically for you when using recent devstack, but if you're deploying via some other method, you need to use python-openstackclient (which is the <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-January/025629.html" target="_blank">only CLI interface to v3 keystone</a>) to create the domain and user:</div>
</div>
<div>
<br /></div>
<div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<b>Create the domain:</b></div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<i>$OS_TOKEN</i> refers to a token, e.g the service admin token or some other valid token for a user with sufficient roles to create users and domains.</div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<i>$KS_ENDPOINT_V3</i> refers to the v3 keystone endpoint, e.g http://<keystone>:5000/v3 where <keystone> is the IP address or resolvable name for the keystone service.</div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<br /></div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"</span><br />
<span style="white-space: pre-wrap;">The domain ID is returned by this command, and is referred to as $HEAT_DOMAIN_ID below.</span><br />
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<br /></div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<b>Create the user:</b></div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 user create --password $PASSWORD --domain $HEAT_DOMAIN_ID heat_domain_admin --description "Manages users and projects created by heat"
</span></div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
The user ID is returned by this command and is referred to as $DOMAIN_ADMIN_ID below:</div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<b><br /></b>
<b>Make the user a domain admin:</b></div>
<div style="white-space: pre-wrap; width: 50em; word-wrap: break-word;">
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 role add --user $DOMAIN_ADMIN_ID --domain $HEAT_DOMAIN_ID admin</span></div>
<pre class="bz_comment_text
bz_wrap_comment_text" id="comment_text_9" style="white-space: pre-wrap; width: 50em; word-wrap: break-word;"><span style="font-family: Courier New, Courier, monospace;">
</span></pre>
<pre class="bz_comment_text
bz_wrap_comment_text" id="comment_text_9" style="white-space: pre-wrap; width: 50em; word-wrap: break-word;"><span style="font-family: 'Times New Roman'; white-space: normal;">Then </span><span style="font-family: 'Times New Roman'; font-size: small; white-space: normal;">you need to add the domain ID, username and password from these steps to heat.conf:</span></pre>
<pre class="bz_comment_text
bz_wrap_comment_text" id="comment_text_9" style="font-size: small; white-space: pre-wrap; width: 50em; word-wrap: break-word;"><span style="font-family: 'Times New Roman'; font-size: small; white-space: normal;">
</span></pre>
<pre class="bz_comment_text
bz_wrap_comment_text" id="comment_text_9" style="width: 50em; word-wrap: break-word;"><div style="width: 50em; word-wrap: break-word;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small; white-space: normal;">stack_domain_admin_password = <password></span></div>
<div style="width: 50em; word-wrap: break-word;">
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small; white-space: normal;">stack_domain_admin = heat_domain_admin</span></div>
<span style="font-size: x-small;"><span style="font-family: Courier New, Courier, monospace;"><span style="white-space: normal;">stack_user_domain = <domain id returned from domain create above></span></span></span>
</pre>
</div>
<h3>
</h3>
<h4>
When a user creates a stack:</h4>
<div>
<ul>
<li>We create a new "stack domain project" in the "heat" domain, if the stack contains any resources which require creation of a "stack domain user"</li>
<li>Any resources which require a user, we create the user in the "stack domain project", which is associated with the heat stack in the heat database, but is completely separate and unrelated (from an authentication perspective) to the stack owners project</li>
<li>The users created in the stack domain are still assigned the heat_stack_user role, so as before the API surface they can access is limited via policy.json</li>
<li>When API requests are processed, we do an internal lookup, and allow stack details for a given stack to be retrieved from the database for both the stack owner's project (the default API path to the stack), and also the "stack domain project", subject to the policy.json restrictions.</li>
</ul>
<div>
To clarify that last point, that means there are now two paths which can result in retrieval of the same data via the heat API, e.g for resource-metadata:</div>
<div>
<span style="background-color: white; color: #535353; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;"><br /></span></div>
<div>
<span style="background-color: white; color: #535353; line-height: 20px;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">GET v1/{stack_owner_project_id}/stacks/{stack_name}/{stack_id}/resources/{resource_name}/metadata</span></span></div>
<div>
<br /></div>
<div>
or</div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="background-color: white; color: #535353; line-height: 20px;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">GET v1/{stack_domain_project_id}/stacks/{stack_name}/{stack_id}/resources/{resource_name}/metadata</span></span></div>
<div>
<span style="background-color: white; color: #535353; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px;"><br /></span></div>
<div>
<span style="background-color: white; color: #535353; line-height: 20px;"><span style="font-family: inherit;">The stack owner would use the former (e.g via "heat resource-metadata {stack_name} {resource_name}), and any agents in the instance will use the latter.</span></span></div>
<div>
<br /></div>
<div>
This solves all of the problems identified previously:</div>
</div>
<div>
<ul>
<li>The stack owner no longer requires admin roles, because the heat_domain_admin user administers stack domain users</li>
<li>There is complete separation, the users created in the stack domain project cannot access any resources other than those explicitly allowed by heat, any attempt to access other stacks, or any other resource owned by the stack-owner will fail.</li>
<li>The list of users in the stack-owner project is unaffected, because we've created a completely different project in another domain.</li>
</ul>
</div>
<div>
Hopefully that provides a fairly clear picture of the new feature, and how it works - it should be transparent to users but I'm hoping this information may be useful to deployers when adopting the functionality for Icehouse.</div>
<div>
<br /></div>
<div>
The main gap still to be investigated is how we handle situations where keystone is backed by a read-only directory (e.g LDAP), my expectation is that it can be solved via the keystone capability to have <a href="https://blueprints.launchpad.net/keystone/+spec/multiple-ldap-servers" target="_blank">different identity drivers per domain</a>, so you could for example have e.g domains containing human users backed by LDAP, and the heat domain backed by SQL. My understanding is that there are <a href="https://blueprints.launchpad.net/keystone/+spec/multi-backend-uuids" target="_blank">outstanding issues to be solved for Juno in keystone</a>, but I will post a future update when I've had time to do some testing and figure out what works.</div>
<div>
<br /></div>
<div>
That is all, respect if you managed to read it all! ;)</div>
<div>
<br /></div>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com38tag:blogger.com,1999:blog-7962934533489004582.post-12867468986217361842014-04-09T09:13:00.000-07:002014-04-10T07:12:55.043-07:00Heat auth model updates - part 1 TrustsOver the last few months I've spent a lot of my time looking at ways to rework the heat auth model, in an attempt to solve two long-standing issues:<br />
<br />
<br />
<ol>
<li>Requirement to pass a password when creating a stack which may perform deferred orchestration actions (for example AutoScaling adjustments)</li>
<li>Requirement for users to have administrative roles when creating certain types of resource.</li>
</ol>
<br />
<br />
So, fixes to these issues have been happening (in <a href="https://blueprints.launchpad.net/heat/+spec/heat-trusts" target="_blank">Havana</a> and <a href="https://blueprints.launchpad.net/heat/+spec/instance-users" target="_blank">Icehouse</a> respectively), but discussions with various folks indicates significant confusion re differentiating the two changes, probably because I've not got around to writing up the documentation yet (it's in progress, honest!) ;)<br />
<br />
In an attempt to clear up the confusion, and provide some documentation ahead of the upcoming <a href="https://blueprints.launchpad.net/heat/icehouse" target="_blank">Icehouse Heat release,</a> I'm planning to cover each feature in this and a subsequent post - below is a discussion of the "Requirement to pass a password" problem, and the method used to solve it.<br />
<br />
<a name='more'></a><br /><br />
<br />
<h3>
What? Passwords? Don't we pass tokens?</h3>
Well, yes mostly we do. However the problem with tokens is they expire, and we have no way of knowing how long a stack may exist for, so we can't store user tokens to do deferred operations after the initial creation of the heat stack (not that it's a good idea from a security perspective either..)<br />
<br />
So in previous versions of heat, we've required the user to pass a password (yes, even if they are passing us a token), which we'd then encrypt and store in the heat database, such that we can then obtain a token to act on behalf of the user and to whatever deferred operations are required during the lifetime of the stack. It's not a nice design, but when it was implemented, <a href="https://blueprints.launchpad.net/keystone/+spec/trusts" target="_blank">Trusts</a> did not exist in Keystone so there was no viable alternative. Here's exactly what happens:<br />
<br />
<br />
<ul>
<li>User requests stack creation, providing a token and username/password (python-heatclient or Horizon normally requests the token for you)</li>
<li>If the stack contains any resources <a href="https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L120" target="_blank">marked as requiring deferred operations</a> heat will fail validation checks if no username/password is provided</li>
<li>The username/password are encrypted and stored in the heat DB</li>
<li>Stack creation is completed</li>
<li>At some later stage we retrieve the credentials and request another token on behalf of the user, the token is not limited in scope and provides access to all roles of the stack owner.</li>
</ul>
<div>
Clearly this is suboptimal, and is the reason for this strange additional password box in horizon:</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgulQxOaPC-VXxGO0h0sNNgIhkWsj0Pb5YZBSVhpHA4yD4LpysgusM5rb03KkrZQVaD226Fa17CLannP0d8zm-c_-7z7YWUoYn7p70wjmAVzKDbHpP-pBWg9DMud1boUAVJtKVR7ErEEMg/s1600/horizon_password.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgulQxOaPC-VXxGO0h0sNNgIhkWsj0Pb5YZBSVhpHA4yD4LpysgusM5rb03KkrZQVaD226Fa17CLannP0d8zm-c_-7z7YWUoYn7p70wjmAVzKDbHpP-pBWg9DMud1boUAVJtKVR7ErEEMg/s1600/horizon_password.png" height="464" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">You already entered your password, right?!</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<br />
Happily, after discussions with Adam Young, Trusts were implemented during <a href="https://blueprints.launchpad.net/keystone/grizzly" target="_blank">Grizzly</a> and Heat integrated with the functionality during the <a href="https://blueprints.launchpad.net/heat/+spec/heat-trusts" target="_blank">Havana</a> cycle. I get the impression not that many people have yet adopted it, so I'm hoping we can move towards making the new <a href="https://bugs.launchpad.net/heat/+bug/1286157" target="_blank">trusts based method the defaul</a>t, which has already <a href="https://review.openstack.org/#/c/80002/" target="_blank">happened for devstack</a> quite recently.<br />
<br />
<br />
<h3>
Keystone Trusts 101</h3>
So, in describing the solution to Heat storing passwords, I will be referring to <a href="https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3-os-trust-ext.md" target="_blank">Keystone Trusts</a>, because that is the method used to implement the solution. There's quite a bit of good information out there, including the <a href="https://wiki.openstack.org/wiki/Keystone/Trusts" target="_blank">Keystone Wiki</a>, <a href="http://adam.younglogic.com/2013/03/trusts-rbac/" target="_blank">Adam Young's blog</a> and the <a href="https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3-os-trust-ext.md" target="_blank">API documentation</a>, but here's a quick summary of terminology which should be sufficient to understand how we're using trusts in Heat:<br />
<br />
Trusts are a keystone extension, which provide a method to enable delegation, and optionally impersonation via keystone. The key terminology is <i>trustor</i> (the user delegating) and <i>trustee</i> (the user being delegated to).<br />
<br />
To create a trust, the <i>trustor</i> (in this case the user creating the heat stack) provides keystone with the following information:<br />
<br />
<br />
<ul>
<li>The ID of the <i>trustee</i> (who you want to delegate to, in this case the heat service user)</li>
<li>The roles to be delegated (configurable via the heat configuration file, but it needs to contain whatever roles are required to perform the deferred operations on the users behalf, e.g launching a nova instance in response to an AutoScaling event)</li>
<li>Whether to enable impersonation</li>
</ul>
<div>
Keystone then provides a <i>trust_id</i>, which can be consumed by the <i>trustee</i> (and <b>only</b> the trustee) to obtain a <i>trust scoped token. </i>This token is limited in scope such that the trustee has limited access to those roles delegated, along with effective impersonation of the trustor user, if it was selected when creating the trust.</div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
Phew! Ok so how did you fix it?</h3>
<div>
Basically we now do the following:</div>
<div>
<br /></div>
<div>
<ul>
<li>User creates a stack via an API request (only the token is required)</li>
<li>Heat uses the token to create a trust between the stack owner (<i>trustor</i>) and the heat service user (<i>trustee), </i>delegating a special role (or roles) as defined in the trusts_delegated_roles list in the heat configuration file. By default heat sets this to "heat_stack_owner", so this role must exist and the user creating the stack must have this role assigned in the project they are creating a stack. Deployers may modify this list to reflect local RBAC policy, e.g to ensure the heat process can only access those services expected while impersonating a stack owner.</li>
<li>Heat stores the trust id in the heat DB (still encrypted, although in theory it doesn't need to be since it's useless to anyone other than the trustee, e.g the heat service user)</li>
<li>When a deferred operation is required, Heat retrieves the trust id, and requests a <i>trust scoped token</i> which enables the service user to impersonate the stack owner for the duration of the deferred operation, e.g to launch some nova instances on behalf of the stack owner in response to an AutoScaling event.</li>
</ul>
<div>
<br /></div>
<div>
The advantages of this approach are hopefully clear, but to clarify:</div>
</div>
<div>
<ul>
<li>It's better for users, we no longer require a password and can provide full functionality when provided with just a token (like all other OpenStack services... and we can kill the Horizon password box, yay!)</li>
<li>It's more secure, as we no longer store any credentials or other data which could use used by any attacker - the trust_id can only be consumed by the <i>trustee</i> (the heat service user).</li>
<li>It provides much more granular control of what can be done by heat in deferred operations, e.g if the stack owner has administrative roles, there's no need to delegate them to Heat, just the subset required.</li>
</ul>
<div>
<br /></div>
<div>
I'd encourage everyone to switch to using this feature, enabling it is simple, first update your <a href="https://github.com/openstack/heat/blob/master/etc/heat/heat.conf.sample#L36" target="_blank">heat.conf file</a> to have the following lines:</div>
</div>
<div>
<br /></div>
<br />
<div>
<span style="font-family: Courier New, Courier, monospace;">deferred_auth_method=trusts</span></div>
<div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">trusts_delegated_roles=heat_stack_owner</span></div>
</div>
<div>
<br /></div>
<div>
Hopefully this will soon become the default from Juno for Heat.</div>
<div>
<br /></div>
<div>
Then ensure all users creating heat stacks have the "heat_stack_owner" role (or whatever roles you want them to delegate to the heat service user based on your local RBAC policies).</div>
<div>
<br /></div>
</div>
<div>
That is all, more coming soon on "<a href="https://blueprints.launchpad.net/heat/+spec/instance-users" target="_blank">stack domain users</a>" which is new for <a href="https://blueprints.launchpad.net/heat/icehouse" target="_blank">Icehouse</a> and resolves the second problem mentioned at the start of this post! :)</div>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com18tag:blogger.com,1999:blog-7962934533489004582.post-3399922616525706612013-10-02T03:51:00.000-07:002013-10-02T06:11:32.176-07:00<h2>
Heat Providers/Environments 101</h2>
<div>
I've recently been experimenting with some cool <a href="https://blueprints.launchpad.net/heat/havana" target="_blank">new features</a> we've added to Heat over the Havana cycle, testing things out in preparation for the <a href="https://wiki.openstack.org/wiki/Havana_Release_Schedule" target="_blank">Havana Release</a>.</div>
<div>
<br /></div>
<div>
One potentially very powerful new abstraction is the <a href="https://blueprints.launchpad.net/heat/+spec/provider-resource" target="_blank">Provider Resource</a> method of defining <a href="http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::CloudFormation::Stack" target="_blank">nested stack resources</a>. Combined with the new <a href="https://wiki.openstack.org/wiki/Heat/Environments" target="_blank">environments</a> capability to map template resource names to non-default implementations, it provides a very flexible way for both users and those deploying Heat to define custom resources based on <a href="http://docs.openstack.org/developer/heat/template_guide/" target="_blank">Heat templates</a>.</div>
<div>
<br /></div>
<div>
Firstly let me clarify what nested stacks are, following on from my <a href="http://hardysteven.blogspot.co.uk/2013/08/heat-nested-resource-introspection.html" target="_blank">previous post</a>, and why we decided to provide this native interface to the functionality, rather than something similar to the existing <a href="http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::CloudFormation::Stack" target="_blank">Cloudformation compatible resource</a> interface.</div>
<div>
<br /></div>
<div>
So nested stack resources enable you to specify a URL of another heat template, which will be used to create another Heat stack, owned by the stack which defines the <a href="http://docs.openstack.org/developer/heat/template_guide/cfn.html#AWS::CloudFormation::Stack" target="_blank">AWS::CloudFormation::Stack</a>. This provides a way to implement composed Heat templates, and to reuse logically related template snippets.</div>
<div>
<br /></div>
<div>
The interface looks like this (using the <a href="http://docs.openstack.org/developer/heat/template_guide/hot_guide.html" target="_blank">new native HOT template syntax</a>):</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">resources:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> nested:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> type: AWS::CloudFormation::Stack</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> properties:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> TemplateURL: http://somewhere/something.yaml</span></div>
</div>
<div>
<br /></div>
<div>
There are several disadvantages to this interface:</div>
<div>
<ul>
<li>Hard-coded URLs in your template</li>
<li>You have to have the nested templates accessible via a web-server somewhere</li>
<li>Hard to transparently substitute different versions of the nested implementation (without sedding! ;)</li>
<li>Passing Parameters is kind-of awkward (pass a nested map of parameter values)</li>
<li>Does not provide a way to define resources based on stack templates.</li>
</ul>
</div>
<div>
So we noticed something, which was the interface to stack templates is essentially very similar to the interface to resources, stack <i>parameters</i> map to resource <i>properties, </i>and stack <i>outputs</i> map to resource<i> attributes</i>. Provider resources leverage this symmetry to provide a more flexible (and arguably easier to use) native interface to nested stack resources.</div>
<div>
<br /></div>
<div>
Ok, so how does it work! Probably the easiest explanation is a simple worked example.</div>
<div>
<br /></div>
<div>
Say you define a simple template, like this <a href="https://github.com/openstack/heat-templates/blob/master/hot/F18/WordPress_Native.yaml" target="_blank">example in our heat-templates repo</a>, and you want to reuse it, as a nested stack via the Providers functionality, you simply need to do this:</div>
<div>
<br /></div>
<h3>
Define an environment</h3>
<div>
The <a href="http://docs.openstack.org/developer/heat/template_guide/environment.html" target="_blank">environment</a> is used to map resource names to templates, and optionally can be used to define common stack parameters, so that they don't need to be passed every time you create a stack.<br />
<br />
A user may pass an environment when creating (or updating) a stack, and a global environment may also be specified by the deployer (default location is /etc/heat/environment.d), using the same syntax. The environment may override existing resources.</div>
<div>
<br /></div>
<div>
The <i>resource_registry </i>section is used to map resource names to template files:</div>
<div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">resource_registry:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> My::WP::Server: https://raw.github.com/openstack/heat-templates/master/hot/F18/WordPress_Native.yaml</span></div>
</div>
<div>
<br /></div>
<div>
Or you can refer to a local file (local to the user running python-heatclient, which reads the file and attaches the content to the Heat API call creating the stack in the "files" parameter of the request):</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">resource_registry:</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> My::WP::Server: file:///home/shardy/git/heat-templates/hot/F18/WordPress_Native.yaml</span></div>
</div>
<br />
There are also some other possibilities, for example aliasing one resource name to another, which are described in <a href="http://docs.openstack.org/developer/heat/template_guide/environment.html" target="_blank">our documentation.</a><br />
<h3>
Create Stack</h3>
<div>
Now you can simply create a stack template which references <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">My::WP::Server:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span></div>
<div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"># cat minimal_test.yaml </span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;">heat_template_version: 2013-05-23</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;">description: ></span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> Heat WordPress template, demonstrating Provider Resource.</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;">parameters:</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> user_key:</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> type: string</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> description : Name of a KeyPair to enable SSH access to the instance</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;">resources:</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> wordpress_instance:</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> type: My::WP::Server</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> properties:</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"> key_name: {get_param: user_key}</span></div>
<div style="font-family: 'Courier New', Courier, monospace; font-size: small;">
<br /></div>
<div>
<span style="font-family: inherit;">With an environment file:</span></div>
<div style="font-size: small;">
<div style="font-size: medium;">
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"># cat env_minimal.yaml </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">resource_registry:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> My::WP::Server: file:///home/shardy/git/heat-templates/hot/F18/WordPress_Native.yaml</span></div>
</div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: inherit;">And then create the stack:</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"># heat stack-create test_stack1 --template-file=./minimal_test.yaml --environment-file=./env_minimal.yaml --parameters="user_key=$USER_key"</span></div>
<div style="font-size: small;">
<br /></div>
<div>
<span style="font-family: inherit;">Optionally you could also specify the key via the environment too:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"></span><br />
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"># cat env_key.yaml </span></div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">
</span>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">parameters:</span></div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">
<div>
user_key: userkey</div>
<div>
resource_registry:</div>
<div>
My::WP::Server: file:///home/shardy/git/heat-templates/hot/F18/WordPress_Native.yaml</div>
</span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="font-size: small;">
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">heat stack-create test_stack2 --template-file=./minimal_test.yaml --environment-file=./env_key.yaml</span></div>
</div>
<div>
<br /></div>
<div>
This would create the nested template with a key_name parameter of "userkey"</div>
<div>
<br /></div>
<div>
So hopefully that provides an overview of this new feature, for more info please see <a href="http://docs.openstack.org/developer/heat/template_guide" target="_blank">our documentation</a> and we're planning to add some example Provider/Environment examples to our <a href="https://github.com/openstack/heat-templates" target="_blank">example template repository</a> soon.</div>
<div>
<br />
Finally, kudos to <a href="http://www.openstack.org/blog/2013/07/open-mic-spotlight-angus-salkeld/" target="_blank">Angus Salkeld</a>, who implemented the majority of this functionality, thanks Angus! :)</div>
<div>
<br /></div>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com29tag:blogger.com,1999:blog-7962934533489004582.post-34038762823579986922013-08-05T10:34:00.002-07:002013-11-27T08:39:33.449-08:00<h2>
<u>Heat Nested Resource Introspection</u></h2>
<div>
The following topic has come up a couple of times lately on IRC, so I thought I'd put down some details describing $subject, in a more permanent place :)</div>
<div>
<br /></div>
<h3>
<u>Nested Stack Resources, Primer/Overview</u></h3>
<div>
So, Heat has a really powerful feature, which is the ability to nest stack definitions, such that one top-level stack definition may recursively define one or more nested stacks.</div>
<div>
<br /></div>
<span style="font-family: inherit;">There are two ways to define a nested stack:</span><br />
<ol>
<li><span style="font-family: inherit;">Explicitly reference a nested stack template in the parent template (via our implementation of the AWS::CloudFormation::Stack resource type, see this <a href="https://github.com/openstack/heat-templates/blob/master/cfn/F17/WordPress_Composed_Instances.template" target="_blank">example template</a>)</span></li>
<li><span style="font-family: inherit;">Create a new resource type, which internally defines a nested stack (an example of this is our </span><a href="https://github.com/openstack/heat/blob/master/heat/engine/resources/loadbalancer.py#L197" style="font-family: inherit;" target="_blank">simple loadbalancer resource</a><span style="font-family: inherit;">, our implementation of the </span><span style="background-color: white; font-family: inherit; line-height: 18px; white-space: pre;">AWS::ElasticLoadBalancing::LoadBalancer resource)</span></li>
</ol>
<span style="line-height: 18px; white-space: pre;"><span style="font-family: inherit;">There is actually a third way (Provider templates), but that's a bleeding-edge feature so I'm not considering it in this post.</span></span><br />
<span style="line-height: 18px; white-space: pre;"><span style="font-family: inherit;"><br /></span></span>
<span style="line-height: 18px; white-space: pre;"><span style="font-family: inherit;">In both cases, what Heat creates internally is a real stack, referenced by the parent stack via a unique ID. Since the Heat API allows you to request details for a specific stack using a stack UUID, that means you can use the heat API introspection operations to access information about the nested stack in the exact same way as you do for the top level stack.</span></span><br />
<span style="line-height: 18px; white-space: pre;"><span style="font-family: inherit;"><br /></span></span>
<br />
<h3>
<span style="line-height: 18px; white-space: pre;"><span style="font-family: inherit;"><u>Worked Example</u></span></span></h3>
<span style="line-height: 18px; white-space: pre;">If I create a stack, I can use various introspection operations, either via the Heat ReST API, or more conveniently via the "heat" CLI tool provided by <a href="https://pypi.python.org/pypi/python-heatclient" target="_blank">python-heatclient</a> (which uses the Heat ReST API):</span><br />
<br />
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">> heat list</span></div>
<div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+--------------------------------------+------------+---------------+----------------------+</span></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| id | stack_name | stack_status | creation_time |</span></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+--------------------------------------+------------+---------------+----------------------+</span></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| faaca636-ed2f-44d9-b228-909c35b37215 | as123 | CREATE_FAILED | 2013-08-05T09:29:49Z |</span></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+--------------------------------------+------------+---------------+----------------------+</span></div>
<div style="font-size: 12px; line-height: 18px; white-space: pre;">
<br /></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: inherit;">I can use the heat introspection operations using either the stack_name (which Heat requires to be unique per tenant) or the unique id interchangeably:</span></div>
<div style="font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; line-height: 18px; white-space: pre;">
<br /></div>
<div style="font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; line-height: 18px; white-space: pre;">
E.g</div>
<div style="font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; line-height: 18px; white-space: pre;">
<br /></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">heat stack-show as123</span></div>
<div style="font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; line-height: 18px; white-space: pre;">
<br /></div>
<div style="line-height: 18px; white-space: pre;">
<span style="font-family: inherit;">provides the exact same information as</span></div>
<div style="font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; line-height: 18px; white-space: pre;">
<br /></div>
</div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">heat stack-show faaca636-ed2f-44d9-b228-909c35b37215</span></div>
<div>
<br /></div>
</div>
<div>
If the stack contains a resource based on a nested stack (or a directly defined nested stack), we can look up the stack ID like this:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">heat resource-list as123</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+--------------------------+-----------------------------------------+-----------------+-------------</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| logical_resource_id | resource_type | resource_status | updated_time </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+--------------------------+-----------------------------------------+-----------------+-------------</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| CfnUser | AWS::IAM::User | CREATE_COMPLETE | 2013-08-05T09:29:52Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| LaunchConfig | AWS::AutoScaling::LaunchConfiguration | CREATE_COMPLETE | 2013-08-05T09:29:56Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| WebServerKeys | AWS::IAM::AccessKey | CREATE_COMPLETE | 2013-08-05T09:29:56Z |</span></div>
<div>
<span style="color: red; font-family: Courier New, Courier, monospace; font-size: xx-small;">| ElasticLoadBalancer | AWS::ElasticLoadBalancing::LoadBalancer | CREATE_FAILED | 2013-08-05T09:40:49Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| MEMAlarmHigh | AWS::CloudWatch::Alarm | INIT_COMPLETE | 2013-08-05T16:57:54Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| MEMAlarmLow | AWS::CloudWatch::Alarm | INIT_COMPLETE | 2013-08-05T16:57:54Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| WebServerGroup | AWS::AutoScaling::AutoScalingGroup | INIT_COMPLETE | 2013-08-05T16:57:54Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| WebServerScaleDownPolicy | AWS::AutoScaling::ScalingPolicy | INIT_COMPLETE | 2013-08-05T16:57:54Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| WebServerScaleUpPolicy | AWS::AutoScaling::ScalingPolicy | INIT_COMPLETE | 2013-08-05T16:57:54Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+--------------------------+-----------------------------------------+-----------------+--------</span></div>
</div>
<div>
<br /></div>
<div>
Here we can see we have a resource "ElasticLoadBalancer", which is of type "AWS::ElasticLoadBalancing::LoadBalancer", which as I mentioned earlier is defined internally via a nested stack, whose ID we can access via the heat resource-show option, which gives details of the specified resource:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">> heat resource-show as123 ElasticLoadBalancer</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+------------------------+--------------------------------</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| Property | Value </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+------------------------+--------------------------------</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| description | |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| links | http://localhost:8004/v1/1938f0707fe04b58b0053040d4a0fe06/stacks/as123/faaca636-ed2f-44d9-b228-909c35b37215/resources/ElasticLoadBalancer |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| | http://localhost:8004/v1/1938f0707fe04b58b0053040d4a0fe06/stacks/as123/faaca636-ed2f-44d9-b228-909c35b37215 |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| logical_resource_id | ElasticLoadBalancer |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><b><span style="color: red;">| physical_resource_id | 60a1ee88-61fe-4bfb-a020-5837e35a42c9 </span></b> |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| required_by | WebServerGroup |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| resource_status | CREATE_FAILED |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| resource_status_reason | Error: Resource create failed: WaitConditionTimeout: 0 of 1 received |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| resource_type | AWS::ElasticLoadBalancing::LoadBalancer |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| updated_time | 2013-08-05T09:40:49Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+------------------------+--------------------------------</span></div>
</div>
<div>
<br /></div>
<div>
Aha! physical_resource_id provides a UUID for the resource, which just so happens to be the UUID of the underlying nested stack ;)</div>
<div>
<br /></div>
<div>
So you can use that UUID to do introspection operations on the nested stack, e.g:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">heat resource-list <span style="color: red;">60a1ee88-61fe-4bfb-a020-5837e35a42c9</span></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+---------------------+------------------------------------------+-----------------+----------------------+</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| logical_resource_id | resource_type | resource_status | updated_time |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+---------------------+------------------------------------------+-----------------+----------------------+</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| CfnLBUser | AWS::IAM::User | CREATE_COMPLETE | 2013-08-05T09:29:52Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| WaitHandle | AWS::CloudFormation::WaitConditionHandle | CREATE_COMPLETE | 2013-08-05T09:29:52Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| latency_watcher | AWS::CloudWatch::Alarm | CREATE_COMPLETE | 2013-08-05T09:29:52Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| CfnLBAccessKey | AWS::IAM::AccessKey | CREATE_COMPLETE | 2013-08-05T09:29:54Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| LB_instance | AWS::EC2::Instance | CREATE_COMPLETE | 2013-08-05T09:30:49Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">| WaitCondition | AWS::CloudFormation::WaitCondition | CREATE_FAILED | 2013-08-05T09:40:49Z |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">+---------------------+------------------------------------------+-----------------+----------------------+</span></div>
</div>
<div>
<br /></div>
<div>
So we can see that the reason the stack failed was the nested stack WaitCondition resource failed (which we already knew from the top level status string, but hopefully you get the point ;)</div>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com2tag:blogger.com,1999:blog-7962934533489004582.post-11311913296330080382013-07-29T04:38:00.000-07:002013-07-29T04:38:03.831-07:00<u>Roadmap for Heat Havana (part 2)</u><br />
<br />
So with havana2 workload and holidays delaying this follow-up post it's probably a bit late to really call this a roadmap, but what follows is a status update and some further details on what we're working on delivering (or have delivered) for Heat's Havana cycle:<br />
<br />
<u>Ceilometer Integration</u><br />
<br />
Some great work has been going on <a href="https://blueprints.launchpad.net/ceilometer/+spec/alarming" target="_blank">adding alarming features to ceilometer</a>, and recently <a href="https://review.openstack.org/#/q/status:merged+project:openstack/heat+branch:master+topic:bp/watch-ceilometer,n,z" target="_blank">some patches have been landing</a> integrating Heat with this alarming capability. This should allow us to move away from maintaining a metric store and alarming functionality inside heat, which will provide a many benefits:<br />
<br />
<br />
<ul>
<li>Align with one openstack metric/alarm solution</li>
<li>Some alarms can use existing hypervisor-level metrics instead of in-instance agent</li>
<li>Allow extensible alarm resources via Provider templates</li>
<li>Removal of heat-engine periodic evaluation tasks (which will allow easier <a href="https://blueprints.launchpad.net/heat/+spec/multiple-engines" target="_blank">engine scale-out</a>)</li>
</ul>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8eTGtT0OXUSdm0tcLIvEFiSgtiZgHJN_VnfZ5V-3UHpIUu9DjXE0eiLw_VDqR8KcGhyphenhyphenK-Y8Fb7aB9YhspNIvUi861CA9aczDK0UKu3aj8hpumvgTCLSefgOupHgGol2BnzMwdsarV3rg/s1600/architecture-cw-metric-collection4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Heat (grizzy) metric collection mechanism" border="0" height="257" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8eTGtT0OXUSdm0tcLIvEFiSgtiZgHJN_VnfZ5V-3UHpIUu9DjXE0eiLw_VDqR8KcGhyphenhyphenK-Y8Fb7aB9YhspNIvUi861CA9aczDK0UKu3aj8hpumvgTCLSefgOupHgGol2BnzMwdsarV3rg/s400/architecture-cw-metric-collection4.png" title="Heat (grizzy) metric collection mechanism" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Heat (grizzy) metric collection mechanism</td></tr>
</tbody></table>
The diagram above illustrates how the metric collection works in grizzly heat - all metric data is collected via a "cfn-push-stats" agent (typically via a cron job defined in the stack template), which requires credentials (a keystone ec2-keypair) to be deployed inside the instance. The metric data is stored in the heat-engine database, and a periodic task evaluates the currently stored data against the alarm thresholds defined in the template. All in all, a crude (but simple) mechanism which has proven sufficient for initial Heat development purposes in the absence of ceilometer metric/alarm functionality.<br />
<br />
The Havana Heat metric collection mechanism will look different, introducing a dependency on the ceilometer service, which can provide access to the hypervisor level statistics, avoiding the in-instance aspect of the method described above for many metric types:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4t5zchGZ8DVVYmKK6lEVU1OycmiXkbki-_dRx7Fgge9R5XxSUi12byTxAqgFFOpGVWgWRKCm3CHL_X6iAidCYXgHfEd6Pg5ZAKYiBiqogU95IfckZ_wxdrO9eNq5TD-KyyAfDI2MxH-U/s1600/architecture-cm-metric-collection-2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Heat (Havana) metric collection/alarms via Ceilometer" border="0" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4t5zchGZ8DVVYmKK6lEVU1OycmiXkbki-_dRx7Fgge9R5XxSUi12byTxAqgFFOpGVWgWRKCm3CHL_X6iAidCYXgHfEd6Pg5ZAKYiBiqogU95IfckZ_wxdrO9eNq5TD-KyyAfDI2MxH-U/s400/architecture-cm-metric-collection-2.png" title="Heat (Havana) metric collection/alarms via Ceilometer" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Heat (Havana) metric collection/alarms via Ceilometer</td></tr>
</tbody></table>
We are also planning to support a compatibility mode (probably for one release cycle) which will allow existing templates using cfn-push-stats to work with the new Ceilometer based alarm mechanism:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid_Szc5kf-75lTCeSqG1b7MhQSEfiIjB3nAdgJ0nLioUyEoO1uGgfLdcdACoMuzJmKBusmbKvl2e5GuaEn5isbWY_ee89rLgqFGfc8j-chncz8YnK83eKx1h6YY6_GN-Ob5SZECu_bMrA/s1600/architecture-cm-metric-collection-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEid_Szc5kf-75lTCeSqG1b7MhQSEfiIjB3nAdgJ0nLioUyEoO1uGgfLdcdACoMuzJmKBusmbKvl2e5GuaEn5isbWY_ee89rLgqFGfc8j-chncz8YnK83eKx1h6YY6_GN-Ob5SZECu_bMrA/s400/architecture-cm-metric-collection-1.png" width="400" /></a></div>
<br />
This should allow existing users of the Heat metric/alarm features time to migrate to the new metric collection method, and also give us time to work out if a Ceilometer tool or agent will be developed which can replace cfn-push-stats (or if cfn-push-stats can be reworked to direct metric data to a Ceilometer API equivalent of PutMetricData), the exact way forward here is still under discussion.<br />
<br />
<u>Keystone Trusts Integration</u><br />
<br />
Work is in-progress to integrate with the <a href="https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3-os-trust-ext.md" target="_blank">Keystone explicit impersonation "Trusts" feature</a> which was added as a v3 API extension for grizzly. The <a href="https://blueprints.launchpad.net/heat/+spec/heat-trusts" target="_blank">initial focus</a> will be to remove the requirement to store encrypted credentials in the Heat DB (which are used for post-create stack actions, for example AutoScaling adjustments), instead we will create a trust token with the minimum possible roles to perform these actions.<br />
<br />
A second thread of this work is to provide an alternative to creating actual keystone users related to the User, AccessKey and WaitConditionHandle resources - because these resource depend on creating an ec2-keypair we need a way to create a keypair from a trust token, which has been <a href="https://blueprints.launchpad.net/keystone/+spec/ec2-keypairs-from-tokens" target="_blank">proposed as a new keystone feature</a>, but not yet implemented. As such it's not yet clear if we'll be able to complete this second step in the Havana time-frame, but we're looking into it! :)<br />
<br />
<u>HOT Progress</u><br />
<br />
Work has been progressing well in delivering the abstractions related to the new HOT DSL, in particular the work related to <a href="https://blueprints.launchpad.net/heat/+spec/provider-resource" target="_blank">Provider resources</a> and <a href="https://blueprints.launchpad.net/heat/+spec/environments" target="_blank">Environments</a> is now largely complete, the initial <a href="https://blueprints.launchpad.net/heat/+spec/hot-hello-world" target="_blank">"hello world" HOT parser implementation</a> has been completed, and work is under-way completing the various additional blueprints required to enable more complex templates to be expressed. It's a huge piece of work, but all those involved are doing a great job pushing things in the right direction.<br />
<br />
<u>And Much More...</u><br />
<u><br /></u>
There is much more that I've not covered here (more stack update improvements, more neutron fixes and functionality, heat standalone mode, converting InstanceGroups to nested stacks, event persistence, to name a few), but that's all I have time for today - hopefully the info above provides some useful context and detail!<br />
<u><br /></u>Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com2tag:blogger.com,1999:blog-7962934533489004582.post-1745651895400297362013-06-20T11:59:00.000-07:002013-06-20T14:11:13.514-07:00<h2>
<u>Roadmap for Heat Havana (part 1)</u></h2>
<div>
It's been quite a while now since the <a href="https://www.openstack.org/summit/portland-2013/" target="_blank">design summit in Portland</a>, and I've been meaning to write some details of the features we discussed at the summit, and in particular those which have appeared now on our <a href="https://blueprints.launchpad.net/heat/havana" target="_blank">plan for Heat's havana development cycle</a>.</div>
<div>
<br /></div>
<div>
What follows are some highlights of what we're working on, or expect to be working on over the next weeks/months. However I'll start with the disclaimer that this plan is a moving target, particularly since we're seeing an increasing number of new contributors whose planned contributions may not yet be captured on the plan, so please keep an eye on the <a href="https://blueprints.launchpad.net/heat/havana" target="_blank">plan in Launchpad</a> for the latest details on what we're aiming to deliver - in other words, this may all change, but here-goes anyway! ;)</div>
<div>
<br /></div>
<h3>
Concurrent Resource Scheduling</h3>
<div>
Inside heat, we create a dependency graph of resources, which we use to determine ordering of all operations (for example create, update, delete) within stack. For grizzly, the order in which these actions happen is determined via topological sorting followed by performing each action in series.</div>
<div>
<br /></div>
<div>
Clearly this is far from ideal when you have large numbers of non-dependent operations (for example creating a large number of grouped instances), so work has been under-way to improve this situation and perform stack operations in parallel where possible. The initial focus of this work is <a href="https://blueprints.launchpad.net/heat/+spec/concurrent-resource-scheduling" target="_blank">resource creation</a>, but the plan is to eventually perform as many stack operations as possible concurrently, making use of the <a href="https://github.com/openstack/heat/blob/master/heat/engine/scheduler.py" target="_blank">new task scheduler</a> that has been developed, which uses <a href="http://www.python.org/dev/peps/pep-3156/#coroutines-and-the-scheduler" target="_blank">coroutine based task scheduling</a>.</div>
<div>
<br /></div>
<div>
Related to this work, we've been discussing ideas with the wider community around <a href="https://wiki.openstack.org/wiki/Heat/TaskSystemRequirements" target="_blank">requirements for workflow and task scheduling in Heat</a>, so that hopefully we can figure out a way to use a common solution across projects with these sorts of requirements.</div>
<h3>
Stack Suspend/Resume</h3>
<div>
<a href="https://blueprints.launchpad.net/heat/+spec/stack-suspend-resume" target="_blank">This feature</a> is aimed at allowing coordinated suspend/resume of a group of resources (including nested resources), such that you can take either an entire stack, or individual resources offline and the resume them (quickly) at some later time.</div>
<div>
<br /></div>
<div>
The idea is to provide access to the some of the underlying capabilities provided by the <a href="http://api.openstack.org/api-ref.html#ext-os-admin-actions" target="_blank">nova admin actions API</a>, but to use the dependency information we have in Heat to do things in the correct order wrt the stack definition. </div>
<div>
<br /></div>
<div>
We will also handle non-instance suspend operations, for example disabling defined alarms, such that suspend/resume can be performed in a non-destructive way.</div>
<div>
<br /></div>
<div>
We may also provide access to other actions in future, so considerable effort has gone into refactoring such that this should be possible with less effort and duplication.</div>
<h3>
Native Template Language (Heat Orchestration Template aka "HOT")</h3>
<div>
So, this ended up being *the* hot topic at the design summit in Portland, we got the message, loud and clear, that there are a lot of users, and potential users, who would like to see an openstack-native (non CFN-compatible) template language develop.</div>
<div>
<br /></div>
<div>
There are two threads to this work - firstly defining the missing logical abstractions (ie what cannot be adequately expressed via the current heat logical model), and secondly the syntax itself. Most of these efforts are captured as dependencies of this <a href="https://blueprints.launchpad.net/heat/+spec/open-api-dsl" target="_blank">umbrella blueprint</a>, and there is the syntax specific <a href="https://blueprints.launchpad.net/heat/+spec/hot-hello-world" target="_blank">"HOT hello world" effort.</a></div>
<div>
<br /></div>
<div>
This is a large and complex piece of work, and the progress made so far has been good, in particular there are recently aspects of the <a href="https://wiki.openstack.org/wiki/Heat/Environments" target="_blank">Environments</a> and <a href="https://wiki.openstack.org/wiki/Heat/Providers" target="_blank">Providers </a>abstractions landing, which will enable the future work to progress. (I'll post again next week with further details on these aspects)</div>
<div>
<br /></div>
<h3>
To be continued...</h3>
<div>
That's all I have time for today, but hopefully provides a taste of what we've got in the pipeline for havana - there's more, much more (ceilometer integration, keystone trusts, native resource types, engine scale-out, concurrent updates, rolling updates, etc etc!), but I'll have to cover those another time! :) </div>
Steve Hardyhttp://www.blogger.com/profile/08425216810383059832noreply@blogger.com0