Complex data transformations with nested Heat intrinsic functions

Disclaimer, what follows is either pretty neat, or pure-evil depending your your viewpoint ;)  But it's based on a real use-case and it works, so I'm posting this to document the approach, why it's needed, and hopefully stimulate some discussion around optimizations leading to a improved/simplified implementation in the future.

The requirement

In TripleO we have a requirement enable composition of different services onto different roles (groups of physical nodes), we need input data to configure the services which combines knowledge of the enabled services, which nodes/role they're running on, and which overlay network each service is bound to.

To do this, we need to input several pieces of data:

1. A list of the OpenStack services enabled for a particular deployment, expressed as a heat parameter it looks something like this:


  EnabledServices:
    type: comma_delimited_list
    default:
      - heat_api
      - heat_engine
      - nova_api
      - neutron_api
      - glance_api
      - ceph_mon

2. A mapping of service names to one of several isolated overlay networks, such as "internal_api" "external" or "storage" etc:


  ServiceNetMap:
    type: json
    default:
      heat_api_network: internal_api
      nova_api_network: internal_api
      neutron_api_network: internal_api
      glance_api_network: storage
      ceph_mon_network: storage

3. A mapping of the network names to the actual IP address (either a single VIP pointing to a loadbalancer, or a list of the IPs bound to that network for all nodes running the service):

  NetIpMap:
    type: json
    default:
      internal_api: 192.168.1.12
      storage: 192.168.1.13

The implementation, step by step

Dynamically generate an initial mapping for all enabled services

Here we can use a nice pattern which combines the heat repeat function with map_merge:

  map_merge:
    repeat:
      template:
        SERVICE_ip: SERVICE_network
      for_each:
         SERVICE: {get_param: EnabledServices}



Step1: repeat dynamically generates lists (including lists of maps as in this case), so we use it to generate a list of maps for every service in the EnabledServices list with a placeholder for the network, e.g:

  - heat_api_ip: heat_api_network
  - heat_engine_ip: heat_engine_network
  - nova_api_ip: nova_api_network
  - neutron_api_ip: neutron_api_network
  - glance_api_ip: glance_api_network
  - ceph_mon_ip: ceph_mon_network

Step2: map_merge combines this list of maps with only one key to one big map for all EnabledServices




  heat_api_ip: heat_api_network
  heat_engine_ip: heat_engine_network 
  nova_api_ip: nova_api_network
  neutron_api_ip: neutron_api_network
  glance_api_ip: glance_api_network
  ceph_mon_ip: ceph_mon_network

Substitute placeholder for the actual network/IP

We approach this in two passes, with two nested map_replace calls (a new function I wrote for newton Heat which can do key/value substitutions on any mapping):



  map_replace:
    - map_replace:

    - heat_api_ip: heat_api_network
      heat_engine_ip: heat_engine_network 
      nova_api_ip: nova_api_network
      neutron_api_ip: neutron_api_network
      glance_api_ip: glance_api_network
      ceph_mon_ip: ceph_mon_network
         - values: {get_param: ServiceNetMap}
     - values: {get_param: NetIpMap}


Step3: The inner map_replace substitutes the placeholder into the actual network provided in the ServiceNetMap mapping, which gives e.g

  heat_api_ip: internal_api
  heat_engine_ip: heat_engine_network
  nova_api_ip: internal_api
  neutron_api_ip: internal_api
  glance_api_ip: storage
  ceph_mon_ip: storage

  
Note that if there's no network assigned in ServiceNetMap for the service, no replacement will occur, so the value will remain e.g heat_engine_network, more on this later..

Step4: the outer map_replace substitutes the network name, e.g internal_api, with the actual VIP for that network provided by the ServiceNetMap mapping, which gives the final mapping of:


  heat_api_ip: 192.168.1.12
  heat_engine_ip: heat_engine_network 
  nova_api_ip: 192.168.1.12
  neutron_api_ip: 192.168.1.12
  glance_api_ip: 192.168.1.13
  ceph_mon_ip: 192.168.1.13

Filter any values we don't want

As you can see we got a value we don't want - heat_engine is like many non-api services in that it's not bound to any network, it only talks to rabbitmq, so we don't have any entry in ServiceNetMap for it.

We can therefore remove any entries which remain in the mapping using the yaql heat function, which is an interface to run yaql queries inside a heat template. 

It has to be said yaql is very powerful, but the docs are pretty sparse (but improving), so I tend to read the unit tests instead of the docs for usage examples.

  yaql:
    expression: dict($.data.map.items().where(isString($[1]) and not $[1].endsWith("_network")))
      data:
        map:

          heat_api_ip: 192.168.1.12
          heat_engine_ip: heat_engine_network 
          nova_api_ip: 192.168.1.12
          neutron_api_ip: 192.168.1.12
          glance_api_ip: 192.168.1.13
          ceph_mon_ip: 192.168.1.13


Step5: filter all map values where the value is a string, and the string ends with "_network" via yaql, which gives:

  heat_api_ip: 192.168.1.12

  nova_api_ip: 192.168.1.12
  neutron_api_ip: 192.168.1.12
  glance_api_ip: 192.168.1.13
  ceph_mon_ip: 192.168.1.13


So, that's it - we now transformed two input maps and a list into a dynamically generated mapping based on the list items! :)

Implementation, completed

Pulling all of the above together, here's a full example (you'll need a newton Heat environment to run this), it combines all steps described above into one big combination of nested intrinsic functions:
 
Edit - also available on github


heat_template_version: 2016-10-14

description: >
  Example of nested heat functions

parameters:
  NetIpMap:
    type: json
    default:
      internal_api: 192.168.1.12
      storage: 192.168.1.13

  EnabledServices:
    type: comma_delimited_list
    default:
      - heat_api
      - nova_api
      - neutron_api
      - glance_api
      - ceph_mon

  ServiceNetMap:
    type: json
    default:
      heat_api_network: internal_api
      nova_api_network: internal_api
      neutron_api_network: internal_api
      glance_api_network: storage
      ceph_mon_network: storage


outputs:
  service_ip_map:
    description: Mapping of service names to IP address for the assigned network
    value:
      yaql:
        expression: dict($.data.map.items().where(isString($[1]) and not $[1].endsWith("_network")))
        data:
          map:
            map_replace:
              - map_replace:
                  - map_merge:
                      repeat:
                        template:
                          SERVICE_ip: SERVICE_network
                        for_each:
                          SERVICE: {get_param: EnabledServices}
                  - values: {get_param: ServiceNetMap}
              - values: {get_param: NetIpMap}