Heat Providers/Environments 101

I've recently been experimenting with some cool new features we've added to Heat over the Havana cycle, testing things out in preparation for the Havana Release.

One potentially very powerful new abstraction is the Provider Resource method of defining nested stack resources.  Combined with the new environments capability to map template resource names to non-default implementations, it provides a very flexible way for both users and those deploying Heat to define custom resources based on Heat templates.

Firstly let me clarify what nested stacks are, following on from my previous post, and why we decided to provide this native interface to the functionality, rather than something similar to the existing Cloudformation compatible resource interface.

So nested stack resources enable you to specify a URL of another heat template, which will be used to create another Heat stack, owned by the stack which defines the AWS::CloudFormation::Stack.  This provides a way to implement composed Heat templates, and to reuse logically related template snippets.

The interface looks like this (using the new native HOT template syntax):

resources:

  nested:

    type: AWS::CloudFormation::Stack

    properties:

      TemplateURL: http://somewhere/something.yaml

There are several disadvantages to this interface:

• Hard-coded URLs in your template
• You have to have the nested templates accessible via a web-server somewhere
• Hard to transparently substitute different versions of the nested implementation (without sedding! ;)
• Passing Parameters is kind-of awkward (pass a nested map of parameter values)
• Does not provide a way to define resources based on stack templates.

So we noticed something, which was the interface to stack templates is essentially very similar to the interface to resources, stack parameters map to resource properties, and stack outputs map to resource attributes.  Provider resources leverage this symmetry to provide a more flexible (and arguably easier to use) native interface to nested stack resources.

Ok, so how does it work!  Probably the easiest explanation is a simple worked example.

Say you define a simple template, like this example in our heat-templates repo, and you want to reuse it, as a nested stack via the Providers functionality, you simply need to do this:

Define an environment

The environment is used to map resource names to templates, and optionally can be used to define common stack parameters, so that they don't need to be passed every time you create a stack.

A user may pass an environment when creating (or updating) a stack, and a global environment may also be specified by the deployer (default location is /etc/heat/environment.d), using the same syntax.  The environment may override existing resources.

The resource_registry section is used to map resource names to template files:

resource_registry:

    My::WP::Server: https://raw.github.com/openstack/heat-templates/master/hot/F18/WordPress_Native.yaml

Or you can refer to a local file (local to the user running python-heatclient, which reads the file and attaches the content to the Heat API call creating the stack in the "files" parameter of the request):

resource_registry:

    My::WP::Server: file:///home/shardy/git/heat-templates/hot/F18/WordPress_Native.yaml

There are also some other possibilities, for example aliasing one resource name to another, which are described in our documentation.

Create Stack

Now you can simply create a stack template which references My::WP::Server:

# cat minimal_test.yaml 

heat_template_version: 2013-05-23

description: >

  Heat WordPress template, demonstrating Provider Resource.

parameters:

  user_key:

    type: string

    description : Name of a KeyPair to enable SSH access to the instance

resources:

  wordpress_instance:

    type: My::WP::Server

    properties:

        key_name: {get_param: user_key}

With an environment file:

# cat env_minimal.yaml 

resource_registry:

    My::WP::Server: file:///home/shardy/git/heat-templates/hot/F18/WordPress_Native.yaml

And then create the stack:

# heat stack-create test_stack1 --template-file=./minimal_test.yaml --environment-file=./env_minimal.yaml --parameters="user_key=$USER_key"

Optionally you could also specify the key via the environment too:

# cat env_key.yaml 

parameters:

    user_key: userkey

resource_registry:

    My::WP::Server: file:///home/shardy/git/heat-templates/hot/F18/WordPress_Native.yaml

heat stack-create test_stack2 --template-file=./minimal_test.yaml --environment-file=./env_key.yaml

This would create the nested template with a key_name parameter of "userkey"

So hopefully that provides an overview of this new feature, for more info please see our documentation and we're planning to add some example Provider/Environment examples to our example template repository soon.

Finally, kudos to Angus Salkeld, who implemented the majority of this functionality, thanks Angus! :)

Heat Nested Resource Introspection

The following topic has come up a couple of times lately on IRC, so I thought I'd put down some details describing $subject, in a more permanent place :)

Nested Stack Resources, Primer/Overview

So, Heat has a really powerful feature, which is the ability to nest stack definitions, such that one top-level stack definition may recursively define one or more nested stacks.

There are two ways to define a nested stack:

1. Explicitly reference a nested stack template in the parent template (via our implementation of the AWS::CloudFormation::Stack resource type, see this example template)
2. Create a new resource type, which internally defines a nested stack (an example of this is our simple loadbalancer resource, our implementation of the AWS::ElasticLoadBalancing::LoadBalancer resource)

There is actually a third way (Provider templates), but that's a bleeding-edge feature so I'm not considering it in this post.

In both cases, what Heat creates internally is a real stack, referenced by the parent stack via a unique ID. Since the Heat API allows you to request details for a specific stack using a stack UUID, that means you can use the heat API introspection operations to access information about the nested stack in the exact same way as you do for the top level stack.

Worked Example

If I create a stack, I can use various introspection operations, either via the Heat ReST API, or more conveniently via the "heat" CLI tool provided by python-heatclient (which uses the Heat ReST API):

> heat list

+--------------------------------------+------------+---------------+----------------------+

| id                                   | stack_name | stack_status  | creation_time        |

+--------------------------------------+------------+---------------+----------------------+

| faaca636-ed2f-44d9-b228-909c35b37215 | as123      | CREATE_FAILED | 2013-08-05T09:29:49Z |

+--------------------------------------+------------+---------------+----------------------+

I can use the heat introspection operations using either the stack_name (which Heat requires to be unique per tenant) or the unique id interchangeably:

E.g

heat stack-show as123

provides the exact same information as

heat stack-show faaca636-ed2f-44d9-b228-909c35b37215

If the stack contains a resource based on a nested stack (or a directly defined nested stack), we can look up the stack ID like this:

heat resource-list as123

+--------------------------+-----------------------------------------+-----------------+-------------

| logical_resource_id      | resource_type                           | resource_status | updated_time       

+--------------------------+-----------------------------------------+-----------------+-------------

| CfnUser                  | AWS::IAM::User                          | CREATE_COMPLETE | 2013-08-05T09:29:52Z |

| LaunchConfig             | AWS::AutoScaling::LaunchConfiguration   | CREATE_COMPLETE | 2013-08-05T09:29:56Z |

| WebServerKeys            | AWS::IAM::AccessKey                     | CREATE_COMPLETE | 2013-08-05T09:29:56Z |

| ElasticLoadBalancer      | AWS::ElasticLoadBalancing::LoadBalancer | CREATE_FAILED   | 2013-08-05T09:40:49Z |

| MEMAlarmHigh             | AWS::CloudWatch::Alarm                  | INIT_COMPLETE   | 2013-08-05T16:57:54Z |

| MEMAlarmLow              | AWS::CloudWatch::Alarm                  | INIT_COMPLETE   | 2013-08-05T16:57:54Z |

| WebServerGroup           | AWS::AutoScaling::AutoScalingGroup      | INIT_COMPLETE   | 2013-08-05T16:57:54Z |

| WebServerScaleDownPolicy | AWS::AutoScaling::ScalingPolicy         | INIT_COMPLETE   | 2013-08-05T16:57:54Z |

| WebServerScaleUpPolicy   | AWS::AutoScaling::ScalingPolicy         | INIT_COMPLETE   | 2013-08-05T16:57:54Z |

+--------------------------+-----------------------------------------+-----------------+--------

Here we can see we have a resource "ElasticLoadBalancer", which is of type "AWS::ElasticLoadBalancing::LoadBalancer", which as I mentioned earlier is defined internally via a nested stack, whose ID we can access via the heat resource-show option, which gives details of the specified resource:

> heat resource-show as123 ElasticLoadBalancer

+------------------------+--------------------------------

| Property               | Value                                                                            

+------------------------+--------------------------------

| description            |                                                                                                                                           |

| links                  | http://localhost:8004/v1/1938f0707fe04b58b0053040d4a0fe06/stacks/as123/faaca636-ed2f-44d9-b228-909c35b37215/resources/ElasticLoadBalancer |

|                        | http://localhost:8004/v1/1938f0707fe04b58b0053040d4a0fe06/stacks/as123/faaca636-ed2f-44d9-b228-909c35b37215                               |

| logical_resource_id    | ElasticLoadBalancer                                                                                                                       |

| physical_resource_id   | 60a1ee88-61fe-4bfb-a020-5837e35a42c9                                                                                                      |

| required_by            | WebServerGroup                                                                                                                            |

| resource_status        | CREATE_FAILED                                                                                                                             |

| resource_status_reason | Error: Resource create failed: WaitConditionTimeout: 0 of 1 received                                                                      |

| resource_type          | AWS::ElasticLoadBalancing::LoadBalancer                                                                                                   |

| updated_time           | 2013-08-05T09:40:49Z                                                                                                                      |

+------------------------+--------------------------------

Aha! physical_resource_id provides a UUID for the resource, which just so happens to be the UUID of the underlying nested stack ;)

So you can use that UUID to do introspection operations on the nested stack, e.g:

heat resource-list 60a1ee88-61fe-4bfb-a020-5837e35a42c9

+---------------------+------------------------------------------+-----------------+----------------------+

| logical_resource_id | resource_type                            | resource_status | updated_time         |

+---------------------+------------------------------------------+-----------------+----------------------+

| CfnLBUser           | AWS::IAM::User                           | CREATE_COMPLETE | 2013-08-05T09:29:52Z |

| WaitHandle          | AWS::CloudFormation::WaitConditionHandle | CREATE_COMPLETE | 2013-08-05T09:29:52Z |

| latency_watcher     | AWS::CloudWatch::Alarm                   | CREATE_COMPLETE | 2013-08-05T09:29:52Z |

| CfnLBAccessKey      | AWS::IAM::AccessKey                      | CREATE_COMPLETE | 2013-08-05T09:29:54Z |

| LB_instance         | AWS::EC2::Instance                       | CREATE_COMPLETE | 2013-08-05T09:30:49Z |

| WaitCondition       | AWS::CloudFormation::WaitCondition       | CREATE_FAILED   | 2013-08-05T09:40:49Z |

+---------------------+------------------------------------------+-----------------+----------------------+

So we can see that the reason the stack failed was the nested stack WaitCondition resource failed (which we already knew from the top level status string, but hopefully you get the point ;)

Roadmap for Heat Havana (part 2)

So with havana2 workload and holidays delaying this follow-up post it's probably a bit late to really call this a roadmap, but what follows is a status update and some further details on what we're working on delivering (or have delivered) for Heat's Havana cycle:

Ceilometer Integration

Some great work has been going on adding alarming features to ceilometer, and recently some patches have been landing integrating Heat with this alarming capability.  This should allow us to move away from maintaining a metric store and alarming functionality inside heat, which will provide a many benefits:

• Align with one openstack metric/alarm solution
• Some alarms can use existing hypervisor-level metrics instead of in-instance agent
• Allow extensible alarm resources via Provider templates
• Removal of heat-engine periodic evaluation tasks (which will allow easier engine scale-out)

Heat (grizzy) metric collection mechanism

The diagram above illustrates how the metric collection works in grizzly heat - all metric data is collected via a "cfn-push-stats" agent (typically via a cron job defined in the stack template), which requires credentials (a keystone ec2-keypair) to be deployed inside the instance.  The metric data is stored in the heat-engine database, and a periodic task evaluates the currently stored data against the alarm thresholds defined in the template.  All in all, a crude (but simple) mechanism which has proven sufficient for initial Heat development purposes in the absence of ceilometer metric/alarm functionality.

The Havana Heat metric collection mechanism will look different, introducing a dependency on the ceilometer service, which can provide access to the hypervisor level statistics, avoiding the in-instance aspect of the method described above for many metric types:

Heat (Havana) metric collection/alarms via Ceilometer

We are also planning to support a compatibility mode (probably for one release cycle) which will allow existing templates using cfn-push-stats to work with the new Ceilometer based alarm mechanism:

This should allow existing users of the Heat metric/alarm features time to migrate to the new metric collection method, and also give us time to work out if a Ceilometer tool or agent will be developed which can replace cfn-push-stats (or if cfn-push-stats can be reworked to direct metric data to a Ceilometer API equivalent of PutMetricData), the exact way forward here is still under discussion.

Keystone Trusts Integration

Work is in-progress to integrate with the Keystone explicit impersonation "Trusts" feature which was added as a v3 API extension for grizzly.  The initial focus will be to remove the requirement to store encrypted credentials in the Heat DB (which are used for post-create stack actions, for example AutoScaling adjustments), instead we will create a trust token with the minimum possible roles to perform these actions.

A second thread of this work is to provide an alternative to creating actual keystone users related to the User, AccessKey and WaitConditionHandle resources - because these resource depend on creating an ec2-keypair we need a way to create a keypair from a trust token, which has been proposed as a new keystone feature, but not yet implemented.  As such it's not yet clear if we'll be able to complete this second step in the Havana time-frame, but we're looking into it! :)

HOT Progress

Work has been progressing well in delivering the abstractions related to the new HOT DSL, in particular the work related to Provider resources and Environments is now largely complete, the initial "hello world" HOT parser implementation has been completed, and work is under-way completing the various additional blueprints required to enable more complex templates to be expressed.  It's a huge piece of work, but all those involved are doing a great job pushing things in the right direction.

And Much More...

There is much more that I've not covered here (more stack update improvements, more neutron fixes and functionality, heat standalone mode, converting InstanceGroups to nested stacks, event persistence, to name a few), but that's all I have time for today - hopefully the info above provides some useful context and detail!

Roadmap for Heat Havana (part 1)

It's been quite a while now since the design summit in Portland, and I've been meaning to write some details of the features we discussed at the summit, and in particular those which have appeared now on our plan for Heat's havana development cycle.

What follows are some highlights of what we're working on, or expect to be working on over the next weeks/months.  However I'll start with the disclaimer that this plan is a moving target, particularly since we're seeing an increasing number of new contributors whose planned contributions may not yet be captured on the plan, so please keep an eye on the plan in Launchpad for the latest details on what we're aiming to deliver - in other words, this may all change, but here-goes anyway! ;)

Concurrent Resource Scheduling

Inside heat, we create a dependency graph of resources, which we use to determine ordering of all operations (for example create, update, delete) within stack.  For grizzly, the order in which these actions happen is determined via topological sorting followed by performing each action in series.

Clearly this is far from ideal when you have large numbers of non-dependent operations (for example creating a large number of grouped instances), so work has been under-way to improve this situation and perform stack operations in parallel where possible.  The initial focus of this work is resource creation, but the plan is to eventually perform as many stack operations as possible concurrently, making use of the new task scheduler that has been developed, which uses coroutine based task scheduling.

Related to this work, we've been discussing ideas with the wider community around requirements for workflow and task scheduling in Heat, so that hopefully we can figure out a way to use a common solution across projects with these sorts of requirements.

Stack Suspend/Resume

This feature is aimed at allowing coordinated suspend/resume of a group of resources (including nested resources), such that you can take either an entire stack, or individual resources offline and the resume them (quickly) at some later time.

The idea is to provide access to the some of the underlying capabilities provided by the nova admin actions API, but to use the dependency information we have in Heat to do things in the correct order wrt the stack definition.  

We will also handle non-instance suspend operations, for example disabling defined alarms, such that suspend/resume can be performed in a non-destructive way.

We may also provide access to other actions in future, so considerable effort has gone into refactoring such that this should be possible with less effort and duplication.

Native Template Language (Heat Orchestration Template aka "HOT")

So, this ended up being *the* hot topic at the design summit in Portland, we got the message, loud and clear, that there are a lot of users, and potential users, who would like to see an openstack-native (non CFN-compatible) template language develop.

There are two threads to this work - firstly defining the missing logical abstractions (ie what cannot be adequately expressed via the current heat logical model), and secondly the syntax itself.  Most of these efforts are captured as dependencies of this umbrella blueprint, and there is the syntax specific "HOT hello world" effort.

This is a large and complex piece of work, and the progress made so far has been good, in particular there are recently aspects of the Environments and Providers abstractions landing, which will enable the future work to progress.  (I'll post again next week with further details on these aspects)

To be continued...

That's all I have time for today, but hopefully provides a taste of what we've got in the pipeline for havana - there's more, much more (ceilometer integration, keystone trusts, native resource types, engine scale-out, concurrent updates, rolling updates, etc etc!), but I'll have to cover those another time! :)