Thursday, 11 September 2014

Using Heat ResourceGroup resources


This has come up a few times recently, so I wanted to share a howto showing how (and how not) to use the group resources in Heat, e.g OS::Heat::ResourceGroup and OS::Heat::AutoScalingGroup.

The key thing to remember when dealing with these resources is that they can multiply any number of resources (expressed as a heat stack), not just individual resources. This is a very cool feature when you get your head around it! :)

Lets go through a worked example, where we use ResourceGroup to create 5 identical servers, each with a cinder volume of the same size attached.


Resource group basics


To create one server with a volume attached, you define the server, a volume, and a volume attachment resource, like this:


Now, lets say you need 5 (or 500) of these identical servers with an attached volume.  What you do *not* want to do is create three groups of resources (Server, Volume and VolumeAttachment), and somehow try to connect them all together.  This is an anti-pattern which will cause you much pain and frustration! :)

Instead, you need to use ResourceGroup to scale out the combination of resources.  Fortunately, Heat makes this very easy to do.  Lets say you call the template above creating one server with attached volume server_with_volume.yaml, you can create 5 identical nested stacks, each containing one server, volume and volume attachment like this:

Note: currently templates referencing nested stack templates can only be launched via python-heatclient (not the Horizon dashboard, a known issue we're working on resolving).

Simply do heat stack-create my_group -f server_with_volume_group.yaml and Heat will create 5 identical servers, attached to 5 identical volumes!

A more complete example related to the fragments above is available here.


Resource groups and provider resources


What's that you say? You don't like the nested stack reference hard-coded template name? No problem! :) You can also make use of the environment to define a provider resource type alias.

Then specify the type alias instead of the template name in the ResourceGroup definition:

This can be lauched like thisheat stack-create my_group2 -f server_with_volume_group.yaml -e env_server_with_volume.yaml

The example will work exactly as before, only different versions of My::Server::WithVolume can easily be substituted, for example if you need a staging workflow where the resource alias is reused across a large number of templates, different versions of the nested template can easily be specified by changing it in one place (the environment).

That is all, for more information, please see the examples in the heat-templates, and this new example which shows how to attach several identical volumes to one server.

24 comments:

  1. Thanks a lot for the informative post. I have one question regarding dependencies. Say that you create a stack using server_with_volume_group.yaml, but with an additional resource shared between each instance in the volume group. This resource cannot be deleted until the ResourceGroup has been cleaned up (think multi-attach to a Cinder volume). Is there a way that I can structure the template to ensure that delete completes on the ResourceGroup before deleting the shared resource (in this particular case an OS::Cinder::Volume)?

    ReplyDelete
  2. How can the nested stack reference things in the parent, for example I want to make 5 servers, but they all need to attach to the 1 subnet I have in the parent template.

    Also are parameters passed into the sub-stack as-is? Does the sub-stack need to define the parameters or does it get them "for free"

    ReplyDelete
  3. The (belated, sorry) answer to both questions above is to create any shared resources in the parent template, then pass a reference in to the nested template. This will result in the correct dependencies on create/update/delete. You create a parameter inside server_with_volume_group.yaml, then pass the value to it via a property (inside the resource_def).

    Example of doing that can be found here: https://github.com/openstack/heat-templates/blob/master/hot/resource_group/resource_group_index_lookup.yaml#L22

    ReplyDelete
  4. I am getting error: ERROR: mapping values are not allowed in this context


    server_with_volume.yaml
    ================================
    heat_template_version: 2013-05-23


    resources:
    instance:
    type: OS::Nova::Server
    properties:
    name: Test
    flavor: m1.tiny
    image: cirros031
    key_name: DemoAccess
    networks:
    - network: 5346fc1d-958b-4881-b73e-8f139957d262
    security_groups:
    - DemoAccess


    test-stack.yml
    ====================
    heat_template_version: 2013-05-23

    resources:
    instance_port:
    type: OS::Neutron::Port
    properties:
    network_id: 5346fc1d-958b-4881-b73e-8f139957d262
    security_groups:
    - DemoAccess
    fixed_ips:
    - subnet_id: bc513101-5b3f-4212-88e7-513628faf93a
    cirros_nodes_nested_stack:
    type: OS::Heat::ResourceGroup
    properties:
    count: 3
    resource_def:
    type: "https://www.dropbox.com/s/gg1qwnm5bqxe02v/server_with_volume.yaml"

    heat stack-create -f test-stack.yml rack1

    ERROR: mapping values are not allowed in this context

    ReplyDelete
  5. Finally..what I want to achieve is to assign unique name to each instance of a resource group like instance-1,instance-2 etc.,..please provide if any working templates are available.

    ReplyDelete
    Replies
    1. You can use the %index% feature to achieve that, it's only available within a Resource Group.

      resources:
      multi:
      type: OS::Heat::ResourceGroup
      properties:
      count: 2
      resource_def:
      type: OS::Nova::Server
      properties:
      name: instance-%index%
      flavor: 2 GB General Purpose v1

      Delete
    2. Thank You for replying. But I am unable to use that index option:

      centos_nodes:
      type: OS::Heat::ResourceGroup
      properties:
      count: { get_param: num_centos_instances }
      resource_def:
      type: OS::Nova::Server
      properties:
      name: instance-%index%
      flavor: m1.small
      image: CentOS
      key_name: DemoAccess
      networks:
      - network: 5346fc1d-958b-4881-b73e-8f139957d262
      security_groups:
      - DemoAccess
      user_data_format: RAW
      user_data: |
      #cloud-config
      final_message: "The system is finally up, after $UPTIME seconds"
      hostname: { get_param: centos_instance_name }
      chpasswd:
      list: |
      root:wizards123
      centos:wizards123
      expire: False
      ssh_pwauth: True

      In horizon UI, it is showing as instance-%index%. It may be because the templates which are available in my system are of version 2012-12-12. Ex:

      heat resource-template OS::Heat::ResourceGroup

      HeatTemplateFormatVersion: '2012-12-12'
      Outputs:
      refs: {Description: A list of resource IDs for the resources in the group, Value: '{"Fn::GetAtt":
      ["ResourceGroup", "refs"]}'}
      Parameters:
      count: {Default: 1, Description: The number of instances to create., MinValue: 1,
      Type: Number}
      resource_def: {Description: Resource definition for the resources in the group.
      The value of this property is the definition of a resource just as if it had
      been declared in the template itself., Type: Json}
      Resources:
      ResourceGroup:
      Properties:
      count: {Ref: count}
      resource_def: {Ref: resource_def}
      Type: OS::Heat::ResourceGroup


      How Can I update my heat modules to use latest version of templates??

      I had installed heat in my controller node using following commands:

      apt-get install heat-api heat-api-cfn heat-engine python-heatclient

      and then changing /etc/heat/heat.conf

      Delete
  6. Can i create a multiple instance in a single stack.
    I don't want nested stack.

    ReplyDelete
    Replies
    1. Hi yes and no.
      yes because you can work with the %index% e.g. for generating networks ports to attach generated resources by name.
      no because e.g. when you need volumes and want to reference in a server object by name to attach a volume it wont work, because the volume_id is required and wont take a name to map a disk.

      So if you dont need additional disks you are finde to construct something, but are limited to the property types of the type you declare in the resource group.

      Delete
  7. I've been spinning my wheels trying to assign specific ports to each VM in my resource Group....I need to do this for IP address management (either assign a specific IP, or assign a port with port_security_disabled).

    To make things a little more complicated I have a ResourceGroup of ResourceGroups. Basically the inner ResourceGroup is to build an active/stanby HA and the outer defines the count of HA pairs.

    e.g
    ServerClusters:
    type: OS::Heat::ResourceGroup
    properties:
    count: 4
    index_var: "%cluster_id%"
    resource_def:
    type: OS::Heat::ResourceGroup
    properties:
    count: 2
    index_var: "%HA_id%"
    resource_def:
    type: OS::Nova::Server
    properties:
    ...
    networks:
    -port [unique port]

    I've tried a number of things
    1) Create a resource group of ports. This works, but I can't figure out how to reference a unique resource in that resource group.

    2) Create a resource_def type Neutron::Port within my inner ResourceGroup. This loads but I don't know how to reference it since there is no resource name to use for get_resource.
    ex:

    ServerClusters:
    type: OS::Heat::ResourceGroup
    properties:
    count: 4
    index_var: "%cluster_id%"
    resource_def:
    type: OS::Heat::ResourceGroup
    properties:
    count: 2
    index_var: "%HA_id%"
    resource_def:
    type: OS::Neutron::Port
    name: "port_%HA_id%_%cluster_id%"
    ...details

    resource_def:
    type: OS::Nova::Server
    properties:
    ...
    networks:
    -port [unique port]

    Any idea? I'm relatively new to openstack but for my active standby HA to work I need those pairs to be aware of each others IPs...

    thanks!

    ReplyDelete
    Replies
    1. @Winston - I think the answer is probably to abstract the inner ResourceGroup to a nested stack template, similar to this example:

      https://github.com/openstack/heat-templates/blob/master/hot/resource_group/resource_group_index_lookup.yaml

      Then you can pass a map of IPs into that nested stack (as a json parameter, which supports maps, unlike the example above which shows using only lists via a comma_delimited_list parameter), and look up the value for each ResourceGroup member via the index_var.

      Delete
  8. So I can see how that works - and it's partially working. Now I run into an issue with a nested call to include my networks.

    I have 2 nova networks:
    my_mgmt_net
    my_service_net

    In my prior config I'd be able to use a simple parameter to Nova to attach to the network:
    type: OS::Nova::Server
    properties:
    ...
    networks:
    -network: my_mgmt_net
    -port:

    This would work - it would successfully access my_mgmt_net (this resource is not defined in my heat template, it's built elsewhere). Now that I try including my Nova configuration in a nested stack as suggested, I run into an error with being unable to access the network resource.

    Know if I need to pass the resource down to the nested yaml config?

    ReplyDelete
    Replies
    1. You should probably put the per-server port resource into a nested stack template along with the server, similar to the server/volume example in this post. Or yes you could pass the ID in from some other parent stack (that's a pattern I'd try to avoid though, as the heat *Group resources work much better if all dependencies of the resources being scaled exist in a nested stack that you just scale out.

      Check https://www.openstack.org/videos/video/heat-beyond-the-basics for more ideas on how to approach scaling and encapsulation with nested stacks.

      Delete
  9. Perfect - I got it going. Thanks Steve!

    ReplyDelete
  10. Hi, Steve

    Thank for your post. Now I want to use the heat API to launch the stack. When using the CLI command on heat client, I could put the main template and inlined node_vlan_template.yaml at the same folder, it works. But via API, How could I provide the content of this node_vlan_template.yaml to heat.

    like below:

    node_vlan_group:
    type: OS::Heat::ResourceGroup
    depends_on: [node_sp_net]
    properties:
    count: {get_param: node_vlan_count}
    resource_def:
    type: node_vlan_template.yaml
    properties:
    vlan_index: '%index%'
    vlan_names: {get_param: node_vlan_names}
    vlan_cidrs4: {get_param: node_cidrs_ipv4}
    vlan_gateways4: {get_param: node_gateways_ipv4}

    ReplyDelete
    Replies
    1. Hi, check out the API docs here:

      https://developer.openstack.org/api-ref/orchestration/v1/?expanded=create-stack-detail

      You can pass the file in the "files" map in the POST body that creates the stack, e.g files: {"node_vlan_template.yaml": "content of node_vlan_template.yaml"}

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Thanks Steve for your quick response.
      I am trying to dump the content of node_vlan_template.yaml into POST body, but the request is not accepted. From the heat-api log, it looks the resource_def.type should only be string. Is there anything wrong?

      JSON response : {"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "Property error: : resources.node_vlan_group.properties.resource_def.type: : Value must be a string",
      ............


      And my POST body looks like:
      {...
      'files': {'node_vlan_template.yaml': ......dumped content........},
      ...
      'template': {......'node_vlan_group': {'depends_on': ['node_sp_net'], 'type': 'OS::Heat::ResourceGroup', 'properties': {'count': {'get_param': 'node_vlan_count'}, 'resource_def': {'type': {'get_file': 'node_vlan_template.yaml'}, .....}

      Delete
    4. You don't need to use get_file inside the type definition of resource_def, see:

      http://docs.openstack.org/developer/heat/template_guide/composition.html#use-the-template-filename-as-type

      Delete
    5. Thanks for pointing this, Steve.
      But now the error is the content of nested template cannot be decoded during validation. I do not know if I miss anything. But the nest template could pass the validation if it is validated separately

      2017-02-15 06:17:06.083 13549 INFO heat.common.urlfetch [req-d12dee9a-24a4-4a69-9036-ae6bb304db87 - admin] Fetching data from node_vlan_template.yaml
      2017-02-15 06:17:06.084 13549 INFO heat.engine.stack [req-d12dee9a-24a4-4a69-9036-ae6bb304db87 - admin] Failed to validate: can't be decoded

      And another thing is in the POST body, I include the "files", "template" and "environment", since some parameters will be included in the "environment".
      Do you have any advice?

      Delete
    6. This comment has been removed by the author.

      Delete
    7. You need to pass the template contents as a string in the files map values, my guess is you're yaml loading the file and passing it as a dict?

      For any further questions I'd suggest asking in #heat on Freenode as we've strayed well off-topic from the blog post content, thanks!

      Delete
    8. Many many Thanks Steve! it works now and you pointed the issue
      Really sorry for the spamming.

      Delete
  11. How do I access the resources inside the resource group?
    I have tried this:

    my_indexed_net_group:
    type: OS::Heat::ResourceGroup
    properties:
    count: 4
    resource_def:
    type: OS::Neutron::Net
    properties:
    name: carrier_net_%index%

    And need to pass each net ID when creating the subnets

    my_indexed_subnet_group:
    type: OS::Heat::ResourceGroup
    depends_on: my_indexed_net_group
    properties:
    count: 4
    resource_def:
    type: OS::Neutron::Subnet
    properties:
    enable_dhcp: True
    cidr: 172.168.%index%.0/24
    ip_version: 4
    name: carrier_subnet_%index%
    network_id: { get_resource: my_indexed_net_group.resource.{%index%}}

    ReplyDelete