Wednesday, 16 April 2014

Heat auth model updates - part 2 "Stack Domain Users"

As promised, here's the second part of my updates on the Heat auth model, following on from part 1 describing our use of Keystone trusts.

This post will cover details of the recently implemented instance-users blueprint, which makes use of keystone domains to contain users related to credentials which are deployed inside instances created by heat.  If you just want to know how the new stuff works, you can skip to the last sections :)

So...why does heat create users at all?

Lets start with a bit of context.  Heat has historically needed to do some or all of the following:
  1. Provide metadata to agents inside instances, which poll for changes and apply the configuration expressed in the metadata to the instance.
  2. Signal completion of some action, typically configuration of software on a VM after it is booted (because nova moves the state of a VM to "Active" as soon as it spawns it, not when heat has fully configured it)
  3. Provide application level status or metrics from inside the instance, e.g to allow AutoScaling actions to be performed in response to some measure of performance or quality of service.
Heat provides API's which enable all of these things, but all of those API's require some sort of authentication, e.g credentials so whatever agent is running on the instance is able to access it.  So credentials must be deployed inside the instance, e.g here's how things work if you're using the heat-cfntools agents:

heat-cfntools agents data-flow with CFN-compatible API's


The heat-cfntools agents use signed requests, which requires an ec2 keypair created via keystone, which is then used to sign requests to the heat cloudformation and cloudwatch compatible API's, which are authenticated by heat via signature validation (which uses the keystone ec2tokens extension).

The problem is, ec2 keypairs are associated with a user.  And we don't want to deploy credentials directly related to the stack owner, otherwise any compromise of the (implicitly untrusted) instance could result in a cascading compromise where an attacker could take control of anything the stack-owning user has permission to access.

I've used cfntools/ec2tokens as an example, but the same issue exists if you use any credential available via keystone (token, username/password) which can be used to authenticate with the heat APIs.

So we need separation/isolation of the credentials deployed in the instance, such that we can limit the access allowed to the minimum necessary to make heat work.  Our first attempt at this did the following:
  • Create a new user in keystone, in the same project as the stack owner (either explicitly in the template via User and AccessKey resources, or for some resources such as WaitConditionHandle and ScalingPolicy we do it internally to obtain an ec2 keypair for generation of a pre-signed URL)
  • Add the "heat stack user" to a special role, default "heat_stack_user" (configurable via the heat_stack_user_role in heat.conf)
  • Limit the API surface accessible to the "heat_stack_user" via policy.json, with the expectation that access to other service's will be restricted in a similar way, or denied completely via network separation/firewall rules.
This approach is flawed, and led to this long-standing bug, there are multiple problems:
  • It requires the user creating the stack to have permissions to create users in keystone, which typically requires administrative roles.
  • It doesn't provide complete separation - even with the policy rules, it's possible a compromised stack could abuse the credentials (for example obtaining metadata for some other stack created by the user in the same project)
  • It clutters the user list for the project with spurious (from the user/operator perspective) users who aren't "real" users, the users are a heat implementation detail, and we're exposing them to the end user.

Hmm, that sounds bad, what's the alternative?

Well, we've been considering that for quite some time ;) multiple solutions were discussed:
  • Delegating a subset of user roles via trusts (rejected because token expiry is not optional, and separation from the stack owner is desired, e.g we don't really want to delegate or impersonate them from the instance, we just need an identity which can be verified as related to the stack)
  • Rolling our own auth mechanism based on some random "token" (some folks were in favour of this, but I'm opposed to it, I think we should stick to orchestration and leverage or improve what's in keystone instead of taking on the burden and security risk of maintaining our own auth scheme)
  • Using the keystone OAuth extension to use OAuth keypairs and signed requests.  (This was rejected due to lack of keystoneclient support, e.g client API and auth middleware, maybe we'll revisit enabling this as an option in some future release).
  • Isolating the in-instance users by creating them in a completely separate heat-specific keystone domain.  This idea was first suggested by Adam Young, as is what we ended up implementing for Icehouse.

"Stack Domain Users", the details..

The new approach is, effectively, an optimisation of the existing implementation.  We encapuslate all stack-defined users (ie users created as a result of things contained in a heat template) in a separate domain, which is created specifically to contain things related only to heat stacks.  A user is created which is the "domain admin", and heat uses that user to manage the lifecycle of the users in the "stack user domain".

There are two aspects of this I'll discuss below, firstly what deployers need to do to enable stack domain users in Heat (Icehouse or later), and secondly what actually happens when you create a stack, and how it addresses the previously identified problems:

When deploying heat:

  • A special keystone domain is created, e.g one called "heat" and the ID is set in the "stack_user_domain" option in heat.conf
  • A user with sufficient permissions to create/delete projects and users in the "heat" domain is created, e.g in devstack a user called "heat_domain_admin" is created, and given the admin role on the heat domain.
  • The username/password for the domain admin user is set in heat.conf (stack_domain_admin and stack_domain_admin_password).  This user administers "stack domain users" on behalf of stack owners, so they no longer need to be admins themselves, and the risk of this escalation path is limited because the heat_domain_admin is only given administrative permission for the "heat" domain.
This is all done automatically for you when using recent devstack, but if you're deploying via some other method, you need to use python-openstackclient (which is the only CLI interface to v3 keystone) to create the domain and user:

Create the domain:
$OS_TOKEN refers to a token, e.g the service admin token or some other valid token for a user with sufficient roles to create users and domains.
$KS_ENDPOINT_V3 refers to the v3 keystone endpoint, e.g http://<keystone>:5000/v3 where <keystone> is the IP address or resolvable name for the keystone service.

openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"
The domain ID is returned by this command, and is referred to as $HEAT_DOMAIN_ID below.

Create the user:
openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 user create --password $PASSWORD --domain $HEAT_DOMAIN_ID heat_domain_admin --description "Manages users and projects created by heat"
The user ID is returned by this command and is referred to as $DOMAIN_ADMIN_ID below:

Make the user a domain admin:
openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 role add --user $DOMAIN_ADMIN_ID --domain $HEAT_DOMAIN_ID admin

Then you need to add the domain ID, username and password from these steps to heat.conf:

stack_domain_admin_password = <password>
stack_domain_admin = heat_domain_admin
stack_user_domain = <domain id returned from domain create above>

When a user creates a stack:

  • We create a new "stack domain project" in the "heat" domain, if the stack contains any resources which require creation of a "stack domain user"
  • Any resources which require a user, we create the user in the "stack domain project", which is associated with the heat stack in the heat database, but is completely separate and unrelated (from an authentication perspective) to the stack owners project
  • The users created in the stack domain are still assigned the heat_stack_user role, so as before the API surface they can access is limited via policy.json
  • When API requests are processed, we do an internal lookup, and allow stack details for a given stack to be retrieved from the database for both the stack owner's project (the default API path to the stack), and also the "stack domain project", subject to the policy.json restrictions.
To clarify that last point, that means there are now two paths which can result in retrieval of the same data via the heat API, e.g for resource-metadata:

GET v1/​{stack_owner_project_id}​/stacks/​{stack_name}​/​{stack_id}​/resources/​{resource_name}​/metadata

or

GET v1/​{stack_domain_project_id}​/stacks/​{stack_name}​/​{stack_id}​/resources/​{resource_name}​/metadata

The stack owner would use the former (e.g via "heat resource-metadata {stack_name} {resource_name}), and any agents in the instance will use the latter.

This solves all of the problems identified previously:
  • The stack owner no longer requires admin roles, because the heat_domain_admin user administers stack domain users
  • There is complete separation, the users created in the stack domain project cannot access any resources other than those explicitly allowed by heat, any attempt to access other stacks, or any other resource owned by the stack-owner will fail.
  • The list of users in the stack-owner project is unaffected, because we've created a completely different project in another domain.
Hopefully that provides a fairly clear picture of the new feature, and how it works - it should be transparent to users but I'm hoping this information may be useful to deployers when adopting the functionality for Icehouse.

The main gap still to be investigated is how we handle situations where keystone is backed by a read-only directory (e.g LDAP), my expectation is that it can be solved via the keystone capability to have different identity drivers per domain, so you could for example have e.g domains containing human users backed by LDAP, and the heat domain backed by SQL.  My understanding is that there are outstanding issues to be solved for Juno in keystone, but I will post a future update when I've had time to do some testing and figure out what works.

That is all, respect if you managed to read it all! ;)

93 comments:

  1. Replies
    1. The effectiveness of IEEE Project Domains depends very much on the situation in which they are applied. In order to further improve IEEE Final Year Project Domains practices we need to explicitly describe and utilise our knowledge about software domains of software engineering Final Year Project Domains for CSE technologies. This paper suggests a modelling formalism for supporting systematic reuse of software engineering technologies during planning of software projects and improvement programmes in Final Year Projects for CSE.

      Software management seeks for decision support to identify technologies like JavaScript that meet best the goals and characteristics of a software project or improvement programme. JavaScript Training in Chennai Accessible experiences and repositories that effectively guide that technology selection are still lacking.

      Aim of technology domain analysis is to describe the class of context situations (e.g., kinds of JavaScript software projects) in which a software engineering technology JavaScript Training in Chennai can be applied successfully

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
    2. Thanks to author for the review of this update, i think it was very informative. And i avice you to post this review on Instagram, where a large amount of users could see it, remember that you can also usualy buy instagram followers in order to quickly boost their number.

      Delete
  2. I should have read this much ealier ;) This is great!

    ReplyDelete
  3. Hi Steve,

    Your article is very helpful to me!

    However, when I created a stack, I saw some errors in /var/log/heat/api-cloudwatch.log:
    2014-11-13 03:40:25.718 1331 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
    2014-11-13 03:40:25.726 1331 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
    2014-11-13 03:40:25.728 1331 INFO heat.api.aws.ec2token [-] Authenticating with http://192.168.122.134:5000/v2.0/ec2tokens
    2014-11-13 03:40:25.729 1331 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.122.134
    2014-11-13 03:40:25.745 1331 INFO heat.api.aws.ec2token [-] AWS authentication failure.

    It seems this error happened when the "/usr/bin/cfn-signal" in my VM tried to send a signal. I also see some errors in /var/log/cfn-push-stats.log in my VM:
    ERROR [2014-11-13 23:20:14,986] 403 AccessDenied
    ERROR [2014-11-13 23:20:14,998]User is not authorized to perform action

    I am not sure if anything wrong in the configuration of my OpenStack env, it would be highly appreciated if you can provide any comments/suggestions.

    My email is: zhq527725@gmail.com, can you please let me know your email so that I can provide more detailed info about the error happened in my env?

    Thanks!

    ReplyDelete
  4. I have the 0.3.1 version of the openstack client and when I try to create the domain I get an invalid command error i.e.

    # openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat --description "Owns users and projects created by heat"
    ERROR: cliff.app Invalid command '--description'

    and if I take the description off I get and unkown command error i.e.

    # openstack --os-token $OS_TOKEN --os-url=$KS_ENDPOINT_V3 --os-identity-api-version=3 domain create heat
    ERROR: cliff.app Unknown command ['$KS_ENDPOINT_V3', 'domain', 'create', 'heat']

    ReplyDelete
    Replies
    1. I had to create the domain using curl i.e.

      curl -s \
      -H "X-Auth-Token: $OS_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{ "domain": { "name": "newdomain"}}' \
      http://localhost:5000/v3/domains | python -mjson.tool

      which I found here https://github.com/openstack/keystone/blob/master/doc/source/api_curl_examples.rst

      after that all the other commands worked.

      Delete
  5. Hi,Hi Steve:
    However, when I created a stack, I saw some errors in /var/log/heat/api-cloudwatch.log:
    2014-11-13 03:40:25.718 1331 INFO heat.api.aws.ec2token [-] Checking AWS credentials..
    2014-11-13 03:40:25.726 1331 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone.
    2014-11-13 03:40:25.728 1331 INFO heat.api.aws.ec2token [-] Authenticating with http://192.168.122.134:5000/v2.0/ec2tokens
    2014-11-13 03:40:25.729 1331 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.122.134
    2014-11-13 03:40:25.745 1331 INFO heat.api.aws.ec2token [-] AWS authentication failure.

    i hava the same prolbem as Zhang Qian's.My email is: wj19901008@gmail.com, can you please let me know your email so that I can provide more detailed info about the error happened in my env?
    Waiting for your great help. Thank you!


    ReplyDelete
  6. Hi Steve,

    I find your blogs about Heat auth model really great, may I translate your blogs and share with my colleges?

    ReplyDelete
  7. Well...

    I translated your blog into Chinese:

    https://github.com/tbbrave/tbbrave.github.io/blob/master/translationwork/Heat_auth_model_shardy02.md

    ReplyDelete
    Replies
    1. Thanks for the translations! Note all content on this blog is CC By-SA : http://creativecommons.org/licenses/by-sa/3.0/

      This means you're free to duplicate the content (including translating and/or modifying it) provided you link to this blog as the original source and disclose if any changes were made. Also you should distribute any derived works via the same licence, thanks! :)

      Delete
    2. Oh I was just thinking you have abandoned this blog....

      I work in Beijing Unitedstack, our team came across a problem so I tried to find out how to solve it.

      I think its better to email then continue commenting..?

      Delete
  8. Good explanation,thanks for writing,it is useful for so many developers
    AWS Online Training

    ReplyDelete
  9. read SEE notes for free including all question paper solved.

    ReplyDelete
  10. This information is impressive; I am inspired by your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.

    best openstack training in chennai | openstack course fees in chennai | openstack certification in chennai | redhat openstack training in chennai

    ReplyDelete
  11. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  12. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  13. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  14. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  15. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  16. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  17. An astounding web diary I visit this blog, it's inconceivably magnificent. Strangely, in this current blog's substance made point of fact and sensible. The substance of information is instructive.
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  18. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  19. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van xả khí ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete

  20. Just seen your Article, it amazed me and surpised me with god thoughts that eveyone will benefit from it. It is really a very informative post for all those budding entreprenuers planning to take advantage of post for business expansions. You always share such a wonderful articlewhich helps us to gain knowledge .Thanks for sharing such a wonderful article, It will be deinitely helpful and fruitful article.
    Thanks
    DedicatedHosting4u.com


    ReplyDelete
  21. An overwhelming web journal I visit this blog, it's unfathomably amazing. Unusually, in this present blog's substance made inspiration driving truth and reasonable. The substance of data is enlightening
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  22. Vanskeligheter( van bi ) vil passere. På samme måte som( van điện từ ) regnet utenfor( van giảm áp ) vinduet, hvor nostalgisk( van xả khí ) er det som til slutt( van cửa ) vil fjerne( van công nghiệp ) himmelen.

    ReplyDelete
  23. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  24. Thank you for sharing us education, please kindly visit mine :D

    AGEN POKER TERPERCAYA
    SITUS POKER ONLINE RESMI

    ReplyDelete
  25. Thanks for sharing useful information.. we have learned so much information from your blog..... keep sharing
    Oracle Fusion HCM Online Training

    ReplyDelete
  26. Soma pill is very effective as a painkiller that helps us to get effective relief from pain. This cannot cure pain. Yet when it is taken with proper rest, it can offer you effective relief from pain.
    This painkiller can offer you relief from any kind of pain. But Soma 350 mg is best in treating acute pain. Acute pain is a type of short-term pain which is sharp in nature. Buy Soma 350 mg online to get relief from your acute pain.

    https://globalonlinepills.com/product/soma-350-mg/


    Buy Soma 350 mg
    Soma Pill
    Buy Soma 350 mg online



    Buy Soma 350 mg online
    Soma Pill
    Buy Soma 350 mg

    ReplyDelete

  27. Class College Education training Beauty teaching university academy lesson  teacher master student  spa manager  skin care learn eyelash extensions tattoo spray

    ReplyDelete
  28. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    best servicenow online training

    ReplyDelete
  29. Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..... azure online training

    ReplyDelete

  30. That is nice article from you , this is informative stuff . Hope more articles from you . I also want to share some information about red hat openstack training and tomcat tutorial

    ReplyDelete
  31. Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    Security Guard License
    Ontario Security License
    Security License Ontario
    Security License

    Thank you..

    ReplyDelete
  32. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    blockchain online training
    best blockchain online training
    top blockchain online training

    ReplyDelete
  33. Thanks for this amazing blog, visit Ogen Infosystem for creative web design and development services at an affordable price.
    Top 5 Website Development Company in Delhi

    ReplyDelete
  34. It is really amazing, Thanks for sharing.
    For any type of certification include BIS Certifications visit our website Bis Consultant in delhi
    .


    We provide the IT and telecom equipment manufacturers to get their products certified from BIS,
    India along Product Testing and Type Approval Services from WPC, India

    ReplyDelete
  35. Water bodies are the main source of transportation for international freight forwarding. Due to this, sea freight company in Delhi,
    visit
    Freight Forwarder in Vietnam
    Shipping Company In India

    ReplyDelete
  36. Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
    blockchain online training
    best blockchain online training
    top blockchain online training

    ReplyDelete
  37. I sometimes visit your blog, find them useful and help me learn a lot, here are some of my blogs you can refer to to support me
    bài thơ ngắn buồn
    phát tờ rơi hà nội
    game bắn cá đổi thưởng
    quay hũ đổi thưởng uy tín
    bán tượng phật tại đồng tháp
    làm bằng b2

    ReplyDelete
  38. good Company Registration in Bangladesh
    Our Call Girls in Jaipur is destined to appoint you in the outstanding enjoyment which never closes.
    English sex video

    ReplyDelete
  39. toptan iç giyim tercih etmenizin sebebi kaliteyi ucuza satın alabilmektir. Ürünler yine orjinaldir ve size sorun yaşatmaz. Yine de bilinen tekstil markalarını tercih etmelisiniz.

    Digitürk başvuru güncel adresine hoşgeldiniz. Hemen başvuru yaparsanız anında kurulum yapmaktayız.

    tutku iç giyim Türkiye'nin önde gelen iç giyim markalarından birisi olmasının yanı sıra en çok satan markalardan birisidir. Ürünleri hem çok kalitelidir hem de pamuk kullanımı daha fazladır.

    nbb sütyen hem kaliteli hem de uygun fiyatlı sütyenler üretmektedir. Sütyene ek olarak sütyen takımı ve jartiyer gibi ürünleri de mevcuttur. Özellikle Avrupa ve Orta Doğu'da çokça tercih edilmektedir.

    yeni inci sütyen kaliteyi ucuz olarak sizlere ulaştırmaktadır. Çok çeşitli sütyen varyantları mevcuttur. iç giyime damga vuran markalardan biridir ve genellikle Avrupa'da ismi sıklıkla duyulur.

    iç giyim ürünlerine her zaman dikkat etmemiz gerekmektedir. Üretimde kullanılan malzemelerin kullanım oranları, kumaşın esnekliği, çekmezlik testi gibi birçok unsuru aynı anda değerlendirerek seçim yapmalıyız.

    iç giyim bayanların erkeklere göre daha dikkatli oldukları bir alandır. Erkeklere göre daha özenli ve daha seçici davranırlar. Biliyorlar ki iç giyimde kullandıkları şeyler kafalarındaki ve ruhlarındaki özellikleri dışa vururlar.

    ReplyDelete
  40. This comment has been removed by the author.

    ReplyDelete
  41. aşk kitapları
    youtube abone satın al
    cami avizesi
    cami avizeleri
    avize cami
    no deposit bonus forex 2021
    takipçi satın al
    takipçi satın al
    takipçi satın al
    takipcialdim.com/tiktok-takipci-satin-al/
    instagram beğeni satın al
    instagram beğeni satın al
    btcturk
    tiktok izlenme satın al
    sms onay
    youtube izlenme satın al
    no deposit bonus forex 2021
    tiktok jeton hilesi
    tiktok beğeni satın al
    binance
    takipçi satın al
    uc satın al
    sms onay
    sms onay
    tiktok takipçi satın al
    tiktok beğeni satın al
    twitter takipçi satın al
    trend topic satın al
    youtube abone satın al
    instagram beğeni satın al
    tiktok beğeni satın al
    twitter takipçi satın al
    trend topic satın al
    youtube abone satın al
    takipcialdim.com/instagram-begeni-satin-al/
    perde modelleri
    instagram takipçi satın al
    instagram takipçi satın al
    takipçi satın al
    instagram takipçi satın al

    ReplyDelete
  42. Thank yo for this post, but it would be better if you make detailed video about it. If you are going to post such video on youtube use this site https://soclikes.com to buy youtube likes for it

    ReplyDelete
  43. I really like and appreciate your post.Really thank you! Fantastic.
    salesforce training
    salesforce online training

    ReplyDelete
  44. Thanks for the information. Your content is the best on the internet so keep updating such informative content!

    moving services near me
    commercial movers near me
    house movers ontario
    self storage company

    ReplyDelete
  45. This comment has been removed by the author.

    ReplyDelete
  46. An investment property is a property that has been purchased for the sole purpose of generating income. Investment properties in Bali Investment properties in Bali can produce a return on investment in the form of rental returns or capital growth. Capital growth occurs when the value of the property has increased over time.

    ReplyDelete
  47. Satya Puri Studio is Bali Construction have worked on historically-inspired homes & villa around the world — including “Lux Villa in Bali; Sukh Sagar Haveli in Jodhpur, Rajasthan; Kahani Paradise in Gokarna. Their work shows inspiration across history and styles, tied together with the goal of enhancing life through good design.

    ReplyDelete
  48. I am eternally grateful for everything you’ve taught me. علی منتظری

    ReplyDelete
  49. Sharetipsinfo is known for providing highly accurate Indian stock market tips which covers Cash tips, F&O intraday tips, Nifty intraday tips, Mcx intraday commodity tips and Share market tips with high accuracy.

    ReplyDelete
  50. Thank you for sharing this information. I read your blog and I can't stop my self to read your full blog. Again Thanks and Best of luck to your next Blog in future.
    바카라사이트

    ReplyDelete
  51. This blog covers some most asked topics in the Azure Data Engineer Interview Questions with fully explained answers and details dp203

    ReplyDelete
  52. Microsoft recently updated a certification named Azure Data Engineer Associate. To get this tag you need to clear one examination named: Exam DP-203: Data Engineering on Microsoft Azure.

    ReplyDelete
  53. Très approprié. Très intéressant, bon travail et merci de partager un si bon blog. Votre article est tellement convaincant que je n'arrête jamais d'en parler. Vous faites du bon travail. Continuez comme ça. Écrire pour nous Technologie


    https://www.totosafeguide.com

    ReplyDelete
  54. Thank you for sharing such a informative information with us. Keep on sharing the blog like this. You can also share your thought here at articlesvibe.site

    ReplyDelete
  55. That is nice article from you , this is informative stuff . Hope more articles from you . I also want to share some information about micro nutrient for plant growth

    ReplyDelete
  56. Aimore Technologies is the best Selenium training institute in Chennai with 6+ years of experience. We are offering online and classroom training.
    Visit Us: Selenium Training in Chennai

    ReplyDelete
  57. The new report by Expert Market Research titled, ‘Global Big Data Technology Market Report and Forecast 2022-2027’, gives an in-depth analysis of the global Big Data Technology Market, assessing the market based on its segments like offerings, deployments, applications, organisation sizes, end-uses, and major regions. The report tracks the latest trends in the industry and studies their impact on the overall market. Organisations’ need to gather and analyse massive volumes of data to get insights and make educated decisions is predicted to drive the global market growth. The market is also expected to benefit from the increased integration of AI and machine learning. The major players in the market are IBM Corporation, Microsoft Corporation, Infosys Limited, Oracle Corporation, and Cloudera, Inc., among others. Expert Market Research (EMR) is a leading market research and business intelligence company, ensuring its clients remain at the vanguard of their industries by providing them with exhaustive and actionable market data through its syndicated and custom market reports. For more information, You can visit our website.

    ReplyDelete
  58. Start your career in the cloud platform with advanced DevOps Training in Chennai from Infycle Technologies. In addition to DevOps, get cloud platform training such as Azure, AWS, and GCP to make your knowledge broader. For getting more offers, call 7504633633 and 7502633633.

    ReplyDelete