Wednesday 16 April 2014

Heat auth model updates - part 2 "Stack Domain Users"

As promised, here's the second part of my updates on the Heat auth model, following on from part 1 describing our use of Keystone trusts.

This post will cover details of the recently implemented instance-users blueprint, which makes use of keystone domains to contain users related to credentials which are deployed inside instances created by heat.  If you just want to know how the new stuff works, you can skip to the last sections :)

So...why does heat create users at all?

Lets start with a bit of context.  Heat has historically needed to do some or all of the following:
  1. Provide metadata to agents inside instances, which poll for changes and apply the configuration expressed in the metadata to the instance.
  2. Signal completion of some action, typically configuration of software on a VM after it is booted (because nova moves the state of a VM to "Active" as soon as it spawns it, not when heat has fully configured it)
  3. Provide application level status or metrics from inside the instance, e.g to allow AutoScaling actions to be performed in response to some measure of performance or quality of service.
Heat provides API's which enable all of these things, but all of those API's require some sort of authentication, e.g credentials so whatever agent is running on the instance is able to access it.  So credentials must be deployed inside the instance, e.g here's how things work if you're using the heat-cfntools agents:

heat-cfntools agents data-flow with CFN-compatible API's


The heat-cfntools agents use signed requests, which requires an ec2 keypair created via keystone, which is then used to sign requests to the heat cloudformation and cloudwatch compatible API's, which are authenticated by heat via signature validation (which uses the keystone ec2tokens extension).

The problem is, ec2 keypairs are associated with a user.  And we don't want to deploy credentials directly related to the stack owner, otherwise any compromise of the (implicitly untrusted) instance could result in a cascading compromise where an attacker could take control of anything the stack-owning user has permission to access.

I've used cfntools/ec2tokens as an example, but the same issue exists if you use any credential available via keystone (token, username/password) which can be used to authenticate with the heat APIs.

So we need separation/isolation of the credentials deployed in the instance, such that we can limit the access allowed to the minimum necessary to make heat work.  Our first attempt at this did the following:
  • Create a new user in keystone, in the same project as the stack owner (either explicitly in the template via User and AccessKey resources, or for some resources such as WaitConditionHandle and ScalingPolicy we do it internally to obtain an ec2 keypair for generation of a pre-signed URL)
  • Add the "heat stack user" to a special role, default "heat_stack_user" (configurable via the heat_stack_user_role in heat.conf)
  • Limit the API surface accessible to the "heat_stack_user" via policy.json, with the expectation that access to other service's will be restricted in a similar way, or denied completely via network separation/firewall rules.
This approach is flawed, and led to this long-standing bug, there are multiple problems:
  • It requires the user creating the stack to have permissions to create users in keystone, which typically requires administrative roles.
  • It doesn't provide complete separation - even with the policy rules, it's possible a compromised stack could abuse the credentials (for example obtaining metadata for some other stack created by the user in the same project)
  • It clutters the user list for the project with spurious (from the user/operator perspective) users who aren't "real" users, the users are a heat implementation detail, and we're exposing them to the end user.

Hmm, that sounds bad, what's the alternative?

Well, we've been considering that for quite some time ;) multiple solutions were discussed:
  • Delegating a subset of user roles via trusts (rejected because token expiry is not optional, and separation from the stack owner is desired, e.g we don't really want to delegate or impersonate them from the instance, we just need an identity which can be verified as related to the stack)
  • Rolling our own auth mechanism based on some random "token" (some folks were in favour of this, but I'm opposed to it, I think we should stick to orchestration and leverage or improve what's in keystone instead of taking on the burden and security risk of maintaining our own auth scheme)
  • Using the keystone OAuth extension to use OAuth keypairs and signed requests.  (This was rejected due to lack of keystoneclient support, e.g client API and auth middleware, maybe we'll revisit enabling this as an option in some future release).
  • Isolating the in-instance users by creating them in a completely separate heat-specific keystone domain.  This idea was first suggested by Adam Young, as is what we ended up implementing for Icehouse.

"Stack Domain Users", the details..

The new approach is, effectively, an optimisation of the existing implementation.  We encapuslate all stack-defined users (ie users created as a result of things contained in a heat template) in a separate domain, which is created specifically to contain things related only to heat stacks.  A user is created which is the "domain admin", and heat uses that user to manage the lifecycle of the users in the "stack user domain".

There are two aspects of this I'll discuss below, firstly what deployers need to do to enable stack domain users in Heat (Icehouse or later), and secondly what actually happens when you create a stack, and how it addresses the previously identified problems:

When deploying heat:

  • A special keystone domain is created, e.g one called "heat" and the ID is set in the "stack_user_domain" option in heat.conf
  • A user with sufficient permissions to create/delete projects and users in the "heat" domain is created, e.g in devstack a user called "heat_domain_admin" is created, and given the admin role on the heat domain.
  • The username/password for the domain admin user is set in heat.conf (stack_domain_admin and stack_domain_admin_password).  This user administers "stack domain users" on behalf of stack owners, so they no longer need to be admins themselves, and the risk of this escalation path is limited because the heat_domain_admin is only given administrative permission for the "heat" domain.
This is all done automatically for you when using recent devstack, but if you're deploying via some other method, you need to use python-openstackclient (which is the only CLI interface to v3 keystone) to create the domain and user:

Create the domain:
$OS_TOKEN refers to a token, e.g the service admin token or some other valid token for a user with sufficient roles to create users and domains.
$KS_ENDPOINT_V3 refers to the v3 keystone endpoint, e.g http://<keystone>:5000/v3 where <keystone> is the IP address or resolvable name for the keystone service.

openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"
The domain ID is returned by this command, and is referred to as $HEAT_DOMAIN_ID below.

Create the user:
openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 user create --password $PASSWORD --domain $HEAT_DOMAIN_ID heat_domain_admin --description "Manages users and projects created by heat"
The user ID is returned by this command and is referred to as $DOMAIN_ADMIN_ID below:

Make the user a domain admin:
openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 role add --user $DOMAIN_ADMIN_ID --domain $HEAT_DOMAIN_ID admin

Then you need to add the domain ID, username and password from these steps to heat.conf:

stack_domain_admin_password = <password>
stack_domain_admin = heat_domain_admin
stack_user_domain = <domain id returned from domain create above>

When a user creates a stack:

  • We create a new "stack domain project" in the "heat" domain, if the stack contains any resources which require creation of a "stack domain user"
  • Any resources which require a user, we create the user in the "stack domain project", which is associated with the heat stack in the heat database, but is completely separate and unrelated (from an authentication perspective) to the stack owners project
  • The users created in the stack domain are still assigned the heat_stack_user role, so as before the API surface they can access is limited via policy.json
  • When API requests are processed, we do an internal lookup, and allow stack details for a given stack to be retrieved from the database for both the stack owner's project (the default API path to the stack), and also the "stack domain project", subject to the policy.json restrictions.
To clarify that last point, that means there are now two paths which can result in retrieval of the same data via the heat API, e.g for resource-metadata:

GET v1/​{stack_owner_project_id}​/stacks/​{stack_name}​/​{stack_id}​/resources/​{resource_name}​/metadata

or

GET v1/​{stack_domain_project_id}​/stacks/​{stack_name}​/​{stack_id}​/resources/​{resource_name}​/metadata

The stack owner would use the former (e.g via "heat resource-metadata {stack_name} {resource_name}), and any agents in the instance will use the latter.

This solves all of the problems identified previously:
  • The stack owner no longer requires admin roles, because the heat_domain_admin user administers stack domain users
  • There is complete separation, the users created in the stack domain project cannot access any resources other than those explicitly allowed by heat, any attempt to access other stacks, or any other resource owned by the stack-owner will fail.
  • The list of users in the stack-owner project is unaffected, because we've created a completely different project in another domain.
Hopefully that provides a fairly clear picture of the new feature, and how it works - it should be transparent to users but I'm hoping this information may be useful to deployers when adopting the functionality for Icehouse.

The main gap still to be investigated is how we handle situations where keystone is backed by a read-only directory (e.g LDAP), my expectation is that it can be solved via the keystone capability to have different identity drivers per domain, so you could for example have e.g domains containing human users backed by LDAP, and the heat domain backed by SQL.  My understanding is that there are outstanding issues to be solved for Juno in keystone, but I will post a future update when I've had time to do some testing and figure out what works.

That is all, respect if you managed to read it all! ;)

38 comments:

  1. Replies
    1. Thanks to author for the review of this update, i think it was very informative. And i avice you to post this review on Instagram, where a large amount of users could see it, remember that you can also usualy buy instagram followers in order to quickly boost their number.

      Delete
  2. I should have read this much ealier ;) This is great!

    ReplyDelete
  3. Hi Steve,

    Your article is very helpful to me!

    However, when I created a stack, I saw some errors in /var/log/heat/api-cloudwatch.log:
    2014-11-13 03:40:25.718 1331 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
    2014-11-13 03:40:25.726 1331 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
    2014-11-13 03:40:25.728 1331 INFO heat.api.aws.ec2token [-] Authenticating with http://192.168.122.134:5000/v2.0/ec2tokens
    2014-11-13 03:40:25.729 1331 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.122.134
    2014-11-13 03:40:25.745 1331 INFO heat.api.aws.ec2token [-] AWS authentication failure.

    It seems this error happened when the "/usr/bin/cfn-signal" in my VM tried to send a signal. I also see some errors in /var/log/cfn-push-stats.log in my VM:
    ERROR [2014-11-13 23:20:14,986] 403 AccessDenied
    ERROR [2014-11-13 23:20:14,998]User is not authorized to perform action

    I am not sure if anything wrong in the configuration of my OpenStack env, it would be highly appreciated if you can provide any comments/suggestions.

    My email is: zhq527725@gmail.com, can you please let me know your email so that I can provide more detailed info about the error happened in my env?

    Thanks!

    ReplyDelete
  4. I have the 0.3.1 version of the openstack client and when I try to create the domain I get an invalid command error i.e.

    # openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"
    ERROR: cliff.app Invalid command '--description'

    and if I take the description off I get and unkown command error i.e.

    # openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat
    ERROR: cliff.app Unknown command ['$KS_ENDPOINT_V3', 'domain', 'create', 'heat']

    ReplyDelete
    Replies
    1. I had to create the domain using curl i.e.

      curl -s \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "domain": { "name": "newdomain"}}' \
      http://localhost:5000/v3/domains | python -mjson.tool

      which I found here https://github.com/openstack/keystone/blob/master/doc/source/api_curl_examples.rst

      after that all the other commands worked.

      Delete
  5. Hi,Hi Steve:
    However, when I created a stack, I saw some errors in /var/log/heat/api-cloudwatch.log:
    2014-11-13 03:40:25.718 1331 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
    2014-11-13 03:40:25.726 1331 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
    2014-11-13 03:40:25.728 1331 INFO heat.api.aws.ec2token [-] Authenticating with http://192.168.122.134:5000/v2.0/ec2tokens
    2014-11-13 03:40:25.729 1331 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.122.134
    2014-11-13 03:40:25.745 1331 INFO heat.api.aws.ec2token [-] AWS authentication failure.

    i hava the same prolbem as Zhang Qian's.My email is: wj19901008@gmail.com, can you please let me know your email so that I can provide more detailed info about the error happened in my env?
    Waiting for your great help. Thank you!


    ReplyDelete
  6. Hi Steve,

    I find your blogs about Heat auth model really great, may I translate your blogs and share with my colleges?

    ReplyDelete
  7. Well...

    I translated your blog into Chinese:

    https://github.com/tbbrave/tbbrave.github.io/blob/master/translationwork/Heat_auth_model_shardy02.md

    ReplyDelete
    Replies
    1. Thanks for the translations! Note all content on this blog is CC By-SA : http://creativecommons.org/licenses/by-sa/3.0/

      This means you're free to duplicate the content (including translating and/or modifying it) provided you link to this blog as the original source and disclose if any changes were made. Also you should distribute any derived works via the same licence, thanks! :)

      Delete
    2. Oh I was just thinking you have abandoned this blog....

      I work in Beijing Unitedstack, our team came across a problem so I tried to find out how to solve it.

      I think its better to email then continue commenting..?

      Delete
  8. Good explanation,thanks for writing,it is useful for so many developers
    AWS Online Training

    ReplyDelete
  9. read SEE notes for free including all question paper solved.

    ReplyDelete
  10. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van xả khí ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete
  11. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van xả khí ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete
  12. Soma pill is very effective as a painkiller that helps us to get effective relief from pain. This cannot cure pain. Yet when it is taken with proper rest, it can offer you effective relief from pain.
    This painkiller can offer you relief from any kind of pain. But Soma 350 mg is best in treating acute pain. Acute pain is a type of short-term pain which is sharp in nature. Buy Soma 350 mg online to get relief from your acute pain.

    https://globalonlinepills.com/product/soma-350-mg/


    Buy Soma 350 mg
    Soma Pill
    Buy Soma 350 mg online



    Buy Soma 350 mg online
    Soma Pill
    Buy Soma 350 mg

    ReplyDelete

  13. Class College Education training Beauty teaching university academy lesson  teacher master student  spa manager  skin care learn eyelash extensions tattoo spray

    ReplyDelete
  14. Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..... azure online training

    ReplyDelete

  15. That is nice article from you , this is informative stuff . Hope more articles from you . I also want to share some information about red hat openstack training and tomcat tutorial

    ReplyDelete
  16. It is really amazing, Thanks for sharing.
    For any type of certification include BIS Certifications visit our website Bis Consultant in delhi
    .


    We provide the IT and telecom equipment manufacturers to get their products certified from BIS,
    India along Product Testing and Type Approval Services from WPC, India

    ReplyDelete
  17. I sometimes visit your blog, find them useful and help me learn a lot, here are some of my blogs you can refer to to support me
    bài thơ ngắn buồn
    phát tờ rơi hà nội
    game bắn cá đổi thưởng
    quay hũ đổi thưởng uy tín
    bán tượng phật tại đồng tháp
    làm bằng b2

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete
  19. aşk kitapları
    youtube abone satın al
    cami avizesi
    cami avizeleri
    avize cami
    no deposit bonus forex 2021
    takipçi satın al
    takipçi satın al
    takipçi satın al
    takipcialdim.com/tiktok-takipci-satin-al/
    instagram beğeni satın al
    instagram beğeni satın al
    btcturk
    tiktok izlenme satın al
    sms onay
    youtube izlenme satın al
    no deposit bonus forex 2021
    tiktok jeton hilesi
    tiktok beğeni satın al
    binance
    takipçi satın al
    uc satın al
    sms onay
    sms onay
    tiktok takipçi satın al
    tiktok beğeni satın al
    twitter takipçi satın al
    trend topic satın al
    youtube abone satın al
    instagram beğeni satın al
    tiktok beğeni satın al
    twitter takipçi satın al
    trend topic satın al
    youtube abone satın al
    takipcialdim.com/instagram-begeni-satin-al/
    perde modelleri
    instagram takipçi satın al
    instagram takipçi satın al
    takipçi satın al
    instagram takipçi satın al

    ReplyDelete
  20. Thank yo for this post, but it would be better if you make detailed video about it. If you are going to post such video on youtube use this site https://soclikes.com to buy youtube likes for it

    ReplyDelete
  21. This comment has been removed by the author.

    ReplyDelete
  22. Très approprié. Très intéressant, bon travail et merci de partager un si bon blog. Votre article est tellement convaincant que je n'arrête jamais d'en parler. Vous faites du bon travail. Continuez comme ça. Écrire pour nous Technologie


    https://www.totosafeguide.com

    ReplyDelete
  23. Hello, thank you for the information. I have been able to obtain new information here. However, I use this page to specialize in some technical problems. I have reloaded the page many times, so I was able to load it correctly. I wonder if your web host was OK.의정부출장샵
    동해출장샵
    삼척출장샵
    속초출장샵
    원주출장샵
    강릉출장샵
    춘천출장샵

    ReplyDelete