Wednesday, 9 April 2014

Heat auth model updates - part 1 Trusts

Over the last few months I've spent a lot of my time looking at ways to rework the heat auth model, in an attempt to solve two long-standing issues:

  1. Requirement to pass a password when creating a stack which may perform deferred orchestration actions (for example AutoScaling adjustments)
  2. Requirement for users to have administrative roles when creating certain types of resource.

So, fixes to these issues have been happening (in Havana and Icehouse respectively), but discussions with various folks indicates significant confusion re differentiating the two changes, probably because I've not got around to writing up the documentation yet (it's in progress, honest!) ;)

In an attempt to clear up the confusion, and provide some documentation ahead of the upcoming Icehouse Heat release, I'm planning to cover each feature in this and a subsequent post - below is a discussion of the "Requirement to pass a password" problem, and the method used to solve it.

What? Passwords? Don't we pass tokens?

Well, yes mostly we do.  However the problem with tokens is they expire, and we have no way of knowing how long a stack may exist for, so we can't store user tokens to do deferred operations after the initial creation of the heat stack (not that it's a good idea from a security perspective either..)

So in previous versions of heat, we've required the user to pass a password (yes, even if they are passing us a token), which we'd then encrypt and store in the heat database, such that we can then obtain a token to act on behalf of the user and to whatever deferred operations are required during the lifetime of the stack.  It's not a nice design, but when it was implemented, Trusts did not exist in Keystone so there was no viable alternative.  Here's exactly what happens:

  • User requests stack creation, providing a token and username/password (python-heatclient or Horizon normally requests the token for you)
  • If the stack contains any resources marked as requiring deferred operations heat will fail validation checks if no username/password is provided
  • The username/password are encrypted and stored in the heat DB
  • Stack creation is completed
  • At some later stage we retrieve the credentials and request another token on behalf of the user, the token is not limited in scope and provides access to all roles of the stack owner.
Clearly this is suboptimal, and is the reason for this strange additional password box in horizon:

You already entered your password, right?!

Happily, after discussions with Adam Young, Trusts were implemented during Grizzly and Heat integrated with the functionality during the Havana cycle.  I get the impression not that many people have yet adopted it, so I'm hoping we can move towards making the new trusts based method the default, which has already happened for devstack quite recently.

Keystone Trusts 101

So, in describing the solution to Heat storing passwords, I will be referring to Keystone Trusts, because that is the method used to implement the solution.  There's quite a bit of good information out there, including the Keystone Wiki, Adam Young's blog and the API documentation, but here's a quick summary of terminology which should be sufficient to understand how we're using trusts in Heat:

Trusts are a keystone extension, which provide a method to enable delegation, and optionally impersonation via keystone.  The key terminology is trustor (the user delegating) and trustee (the user being delegated to).

To create a trust, the trustor (in this case the user creating the heat stack) provides keystone with the following information:

  • The ID of the trustee (who you want to delegate to, in this case the heat service user)
  • The roles to be delegated (configurable via the heat configuration file, but it needs to contain whatever roles are required to perform the deferred operations on the users behalf, e.g launching a nova instance in response to an AutoScaling event)
  • Whether to enable impersonation
Keystone then provides a trust_id, which can be consumed by the trustee (and only the trustee) to obtain a trust scoped token.  This token is limited in scope such that the trustee has limited access to those roles delegated, along with effective impersonation of the trustor user, if it was selected when creating the trust.

Phew! Ok so how did you fix it?

Basically we now do the following:

  • User creates a stack via an API request (only the token is required)
  • Heat uses the token to create a trust between the stack owner (trustor) and the heat service user (trustee), delegating a special role (or roles) as defined in the trusts_delegated_roles list in the heat configuration file.  By default heat sets this to "heat_stack_owner", so this role must exist and the user creating the stack must have this role assigned in the project they are creating a stack.  Deployers may modify this list to reflect local RBAC policy, e.g to ensure the heat process can only access those services expected while impersonating a stack owner.
  • Heat stores the trust id in the heat DB (still encrypted, although in theory it doesn't need to be since it's useless to anyone other than the trustee, e.g the heat service user)
  • When a deferred operation is required, Heat retrieves the trust id, and requests a trust scoped token which enables the service user to impersonate the stack owner for the duration of the deferred operation, e.g to launch some nova instances on behalf of the stack owner in response to an AutoScaling event.

The advantages of this approach are hopefully clear, but to clarify:
  • It's better for users, we no longer require a password and can provide full functionality when provided with just a token (like all other OpenStack services... and we can kill the Horizon password box, yay!)
  • It's more secure, as we no longer store any credentials or other data which could use used by any attacker - the trust_id can only be consumed by the trustee (the heat service user).
  • It provides much more granular control of what can be done by heat in deferred operations, e.g if the stack owner has administrative roles, there's no need to delegate them to Heat, just the subset required.

I'd encourage everyone to switch to using this feature, enabling it is simple, first update your heat.conf file to have the following lines:


Hopefully this will soon become the default from Juno for Heat.

Then ensure all users creating heat stacks have the "heat_stack_owner" role (or whatever roles you want them to delegate to the heat service user based on your local RBAC policies).

That is all, more coming soon on "stack domain users" which is new for Icehouse and resolves the second problem mentioned at the start of this post! :)


  1. Thanks for writing this down, Steven! Very helpful!

  2. Awesome ! It helped me to understand the auth model

  3. if we are upgrading from IceHouse to Juno in our development environment and we have stacks created and convert over to trusts by modifying the heat.conf will those stacks still operate if we delete their AD password out of the database?

    1. It doesn't look like you ever got a response to this, did you happen to reach a conclusion on your own? I'm am trying to determine whether it is worth it to upgrade to a release that supports keystone trusts before creating any stacks, please let me know if you ran into troubles with removing the credentials from the heat database in place of trusts as this will help me to make an informed decision on whether to hold off and upgrade first or proceed as is and modify as needed later. Thanks!

    2. Belated response (apologies, I monitor mailing list and IRC traffic more than these comments) - the trusts integration in heat was designed to fall back to the old password based mechanism when no trust is stored for a stack (e.g it was created before an upgrade to a heat where trusts is enabled).

      This means old stacks will not work if you delete the underlying passwords, but they should continue to work provided the credentials aren't removed. You don't switch over to trusts unless you delete and re-create the stack (e.g all new stacks after the upgrade use trusts, old ones are left alone).

      This approach was chosen to avoid excess complexity in the code, but it'd probably be possible to write a translation tool, or even switch stacks over to use trusts on stack-update, I just did not have time tackle those problems when doing the initial heat trusts integration.

  4. Hi Steve,

    Your blogs are really great!
    And to share this blog with my colleges, I translated this into Chinese:

    1. Thanks for the translations! Note all content on this blog is CC By-SA :

      This means you're free to duplicate the content (including translating and/or modifying it) provided you link to this blog as the original source and disclose if any changes were made. Also you should distribute any derived works via the same licence, thanks! :)

  5. Hi Steve,

    Thanks for amazing information. But I want to know one thing, if I want to use deferred_auth_method as password then what more parameters need to set (or pass).

    Because, I am getting this error when I am setting it:
    ERROR: Missing required credential: X-Auth-User

  6. Thank you for sharing us education, please kindly visit mine :D


  7. CrownQQ Agen DominoQQ BandarQ dan Domino99 Online Terbesar

    Yuk Buruan ikutan bermain di website CrownQQ
    Sekarang CROWNQQ Memiliki Game terbaru Dan Ternama loh...

    10 permainan :
    => Poker
    => Bandar Poker
    => Domino99
    => BandarQ
    => AduQ
    => Sakong
    => Capsa Susun
    => Bandar 66
    => Perang Baccarat
    => Perang Dadu

    Keunggulan bermain di CrownQQ :
    => Bonus Refferal 20%
    => Bonus Turn Over 0,5%
    => Minimal Depo 20.000
    => Minimal WD 20.000
    => 100% Member Asli
    => Pelayanan DP & WD 24 jam
    => Livechat Kami 24 Jam Online
    => Bisa Dimainkan Di Hp Android
    => Di Layani Dengan 5 Bank Terbaik
    => 1 User ID 10 Permainan Menarik
    => Menyediakan deposit Via Pulsa

    Link Resmi CrownQQ:

    Info Lebih lanjut Kunjungi :
    Website : DominoQQ Online
    Daftar CrownQQ : Poker Online
    Info CrownQQ : Kontakk
    Linktree : Agen Poker Online

    WHATSAPP : +6287771354805
    Line : CS_CROWNQQ
    Facebook : CrownQQ Official
    Kemenangan CrownQQ : Agen BandarQ

  8. Nagaqq Yang Merupakan Agen Bandarq terbaik , Domino 99, Dan Bandar Poker Online Terpercaya di asia hadir untuk anda semua dengan permainan permainan menarik dan bonus menarik untuk anda semua

    Bonus yang diberikan NagaQQ :
    * Bonus rollingan 0.5%,setiap senin di bagikannya
    * Bonus Refferal 10% + 10%,seumur hidup
    * Bonus Jackpot, yang dapat anda dapatkan dengan mudah
    * Minimal Depo 15.000
    * Minimal WD 20.000
    * Deposit via Pulsa TELKOMSEL

    Memegang Gelar atau title sebagai AGEN POKER ONLINE Terbaik di masanya

    11 Games Yang di Hadirkan NagaQQ :
    * Poker Online
    * BandarQ
    * Domino99
    * Bandar Poker
    * Bandar66
    * Sakong
    * Capsa Susun
    * AduQ
    * Perang Bacarrat
    * Perang Dadu
    * BD QQ (New Game)

    Info Lebih lanjut Kunjungi :
    Website : NAGAQQ
    Facebook : NagaQQ official
    WHATSAPP : +855977509035
    Line : Cs_nagaQQ
    TELEGRAM :+855967014811

    Winner NagaQQ
    Daftar NagaQQ

  9. köp ketamin
    comprare la ketamina
    ketamint vásárolni
    kupiti ketamin
    Ketamin kaufen
    comprar cetamina
    pirkti ketamino
    osta ketamiini
    osta ketamiini veebis
    pirkti ketamino internete
    comprar cetamina online
    kup ketaminę online
    Ketamin online kaufen
    comprar cetamina online
    купить кетамин онлайн
    kjøp ketamin online
    köp ketamin online
    buy online ketamine
    buy ketamine
    order ketamine online
    ut ketamine online
    beställ ketamin online
    bestill ketamin online
    заказать кетамин онлайн
    Ketamin online bestellen
    online ketamine bestellen
    pedir ketamina en línea

    call/text/whatsapp<<<<<<<< +1(505)257-5355


    Yang Merupakan Agen Bandarq, Domino 99, Dan Bandar Poker Online Terpercaya di asia hadir untuk anda semua dengan permainan permainan menarik dan bonus menarik untuk anda semua

    Bonus yang diberikan NagaQQ :
    * Bonus rollingan 0.5%,setiap senin di bagikannya
    * Bonus Refferal 10% + 10%,seumur hidup
    * Bonus Jackpot, yang dapat anda dapatkan dengan mudah
    * Minimal Depo 15.000
    * Minimal WD 20.000

    Games Yang di Hadirkan NagaQQ :
    * Poker Online
    * BandarQ
    * Domino99
    * Bandar Poker
    * Bandar66
    * Sakong
    * Capsa Susun
    * AduQ
    * Perang Bacarrat
    * Perang Dadu
    * BD Q
    * Adu Sakong
    Menyediakan deposit Via E-Money
    Seperti : OVO, Dana , Gopay, Link aja ,dll

    Info Lebih lanjut Kunjungi :
    Website : NagaQQ
    Facebook : NagaQQ Official
    Kontakk : Info NagaQQ
    linktree : Agen Judi Online
    WHATSAPP : +6287833350051
    Line : Cs_nagaQQ
    TELEGRAM : +855967014811

  11. I appreciate the valuable information you provided. Thank you for sharing this wonderful post.- LBM Blockchain Solutions

  12. The way you explained your story is impressive, I think every blogger has to write content in this way. I have also written content on one topic that, I would love to share with you. my blog is related to ICO Token Development services in India. please read the blog and share your review with us.