Create subscriptions programmatically using Azure Resource Manager
Closed this issue · 63 comments
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Description
Create subscriptions programmatically using Azure Resource Manager. Here are the details about the API being released from Azure https://docs.microsoft.com/en-gb/azure/azure-resource-manager/programmatically-create-subscription
New or Affected Resource(s)
Subscriptions
- azurerm_XXXXX
Potential Terraform Configuration
# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.References
- #0000
Hey team, what has blocked the progress of this one?
@a138076 whilst we (or someone else) may be able to build support for this using the SDK - unfortunately we've current got no means of testing this (since we currently don't have an Enterprise Account, which is a requirement here).
@grayzu has been trying to determine if we can get an EA account (and a such I'm going to assign this to him) - however that's why this is blocked at this time.
I have an EA Account and happy to help if I can
I am also interested in this capability. Do you have any reccomendations for how to do this while we wait for the functionallity?
Currently this article seems to describe a possible workaround?
https://thegrayzone.co.uk/blog/2017/03/external-terraform-provider-powershell/
This is high priority feature for us. How can we move this along?
I have access to an Enterprise Account and am trying to solve this issue of creating subscriptions right now. I am happy to facilitate any testing that is needed. Please feel free to reach out to me directly.
👋
To give an update here: after having spent some time investigating this, whilst there's now an SDK available for this (and there's an SDK for Rename/Cancel coming, but I can't find the link right now) - I don't believe it's possible to implement this at this point in time.
At this point in time the only API available for this allows creating a maximum of 50 subscriptions (total) and only for Enrollment Accounts. As we run acceptance tests for all resources in the Provider, we need to be able to create and destroy Subscriptions dynamically every test (which we run every night), thus we'd quickly hit our limit (and be unable to provision any more). In addition for customers, I'm not necessarily sure how useful it would be to only create 50 subscriptions, ever - since you can't also create Enrollment Accounts through the API.
All that to say - after chatting with Microsoft it sounds like they're working to remove that limitation - but there's not a (public) timeframe for that. As the SDK is available for this it should be fairly easy to add support for this once the API limitation's been removed - but since this is currently blocked on a fix in the Azure API I'm going to assign this to the Blocked milestone for the moment - but once the limitation's been removed we'll circle around and pick this up.
Thanks!
Did anyone get feedback on whether the limitation on Azure EA account subscription creation have been lifted by MS?
Looks like there has been some changes now:
There's a limit of 200 subscriptions per enrollment account. After that, more subscriptions for the account can only be created in the Azure portal. If you want to create more subscriptions through the API, create another enrollment account. https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/programmatically-create-subscription?tabs=rest
progress I guess, although this won't resolve the issue at hand :(
Could this be added without an acceptance test for this resource? We only create 1 or 2 subscriptions a month on average a limit of 200 would last us several years.
Very interested in this feature! Thank you to those who are working on pushing it forward.
@chrheg thanks for the heads up - 200 is better than 50, but it's still fundamentally the same problem unfortunately
@adarobin unfortunately not - in practice the acceptance tests have caught a ton of issues for us when they're run both nightly and when we upgrade the Azure SDK (since the SDK is auto-generated and contains no tests in itself) - as such it's a requirement for shipping any resource to be able to confirm it works/continue shipping it.
Whilst I appreciate it's frustrating this feature is blocked - it looks like progress is being made in the Azure API - I'll reach out to Microsoft on our end to see if there's any public timeframe for this too.
Thanks!
@tombuildsstuff what's the issue with only being able to create 200? can't you cancel them?
I would assume you aren't creating 200+ in parallel?
@timja you can create 200 subscriptions through the API, periodg (which still count even if they're cancelled) - rather than concurrently/a quota unfortunately. So running say a couple of tests nightly plus during SDK upgrades (and development) means we'd be unable to run the tests/provision any more subscriptions after a few months - which is the issue here
Once it's possible to provision these on demand this should be fine - but for the moment unfortunately the limit of creating 200 subscriptions (ever) means this isn't something we can ship until that's fixed unfortunately.
oh, I didn't realise, completely understand with that limitation then.
Okay, so you have to use a new service principal every 200 subscription creations, or am I missing something here?
Even though this is kind of annoying, it would be a huge improvement if we could create subscriptions within a terraform deployment.
Okay, so you have to use a new service principal every 200 subscription creations, or am I missing something here?
Even though this is kind of annoying, it would be a huge improvement if we could create subscriptions within a terraform deployment.
No you need a new enrolment account AFAIK
No you need a new enrolment account AFAIK
Correct - and my understanding (although I might be wrong) is that's paperwork to create a new enrollment account rather than an API, as such I don't think we can create those via an API either?
No you need a new enrolment account AFAIK
Correct - and my understanding (although I might be wrong) is that's paperwork to create a new enrollment account rather than an API, as such I don't think we can create those via an API either?
Not that I'm aware of, I think they can just be created in the EA portal, I have my own enrolment account created for me by my company.
But I don't have access to create new ones so can't be sure
Would be great to have this feature in Terraform!
Looking at MSDN docs and it looks like current limit was increased to 500.
@tombuildsstuff I escalated this through our engineering team and they have raised the limit to 500. Could we get this implemented as soon as possible? Please reach out to me directly on Github and we can connect offline.
Here is a link to the APIs for subscriptions. The upstream Azure/azure-sdk-for-go module will need updated for 2019-03-01-preview. The biggest ongoing issue is when you cancel in an EA (not sure if this is true for Pay-as-you-go or CSP), azure holds the subscription for 90 days. See information here. I don't know if this can be changed. I assume we would want to do both EA and Pay-as-you-go (and CSP?).
We are Azure EA customers and I can confirm, that azure holds our subscriptions for 90 days after deletion. Actually this is the desired behavior from our point of view. I think from terraform view we could treat a subscriptions on hold the same way, as a deleted subscription (e.g. as if it does not exist).
Or am I missing something?
Btw. we're really looking forward for azurerm provider to support subscription creation :)
We are Azure EA customers and I can confirm, that azure holds our subscriptions for 90 days after deletion. Actually this is the desired behavior from our point of view. I think from terraform view we could treat a subscriptions on hold the same way, as a deleted subscription (e.g. as if it does not exist).
Or am I missing something?
Btw. we're really looking forward for azurerm provider to support subscription creation :)
Yes we still want this behavior. But if you are running acceptance tests it becomes and issue when you add and remove it 50 times a night and then you exhaust your supply in 10 days.
If people do want to implement this in their own customised provider, dnaeon/go-vcr might be the way you'd test this, otherwise you're going to exhuast your pool of account creation pretty quickly...
Ok, I see, that's in fact a problem I didn't think about 😇
👋
To give an update from our side here: based on the quota's that exist today (essentially 500 subscriptions/90 days) - unfortunately this quota still isn't enough for us to be able to ship and test this - but we're getting close.
To give some math here: allowing for 50 subscriptions / test runs to get this built & merged - we've then got (say) 5 tests a night, plus additional times we run the tests (e.g. SDK upgrades), which is over 500 for a 90 day period, but not by much.
Based on the back-of-the-envelope math above, we're getting close to where this is possible - once the limit reaches (say) 1000/2000 we should be able to take another look at this. From our side there's additional tests we'd need to add over time (e.g. permissions/configuration at a subscription level), so a fixed limit this low isn't ideal - but this should allow us to at least ship this resource.
As mentioned above we're working with Microsoft to make this happen - we're also running down the internal testing related things to make this happen - so we're hoping that by the time these limits get a little higher we should be in a position to take another look/ship this 👍
Thanks!
@petems unfortunately the nature of the Azure API's means the VCD testing wouldn't be sufficient here
Okay, quick question, I thought the limit of 500 subscriptions is "subscription creation through API" as in "one enrollment account can call the create subscription endpoint 500 times" and not "one enrollment account can have up to 500 subscriptions created via API in parallel". Is this assumption wrong? 🤔
Hi Tom. I have been emailing with Katie on this issue to help you get a quota increase for you Azure sub. Can you reach out to Katie and reach me on that thread? thanks!
Hi @tombuildsstuff, @wilbug1git1, do you think you guys will be able to make progress on it together? Most of today's customers have EA account so it's actually a nice feature. Thanks for the great job you guys doing!
We are Azure EA customers and I can confirm, that azure holds our subscriptions for 90 days after deletion. Actually this is the desired behavior from our point of view. I think from terraform view we could treat a subscriptions on hold the same way, as a deleted subscription (e.g. as if it does not exist).
Or am I missing something?
We have been testing this internally since v42.3.0 release of Azure SDK for Go.
Example resource for creating subscription in enrollment account:
https://github.com/innovationnorway/terraform-provider-azurerm/commit/cd40af6f23a59726628ec9147f3f1d947e0c53f6
Our only concern so far, besides from the subscription limit already mentioned, is that creating and deleting (cancelling) subscriptions requires the RBAC Owner role both on enrollment account (ref. https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/grant-access-to-create-subscription?tabs=rest%2Crest-2) and subscription (ref. https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/cancel-azure-subscription#who-can-cancel-a-subscription). However this might not be a concern for most 😄
Looks like the limit was raised to 2000 subscriptions per enrollment account, is that sufficient and/or has there been any possibility to progress this further?
Can't wait for this functionality to be implemented asap!
Can anyone advise at this point in time does the current update only support EA subscriptions?
From what I understand the 2000 subscription limit is now in place, are there any other blockers that are preventing this moving forwards and being included in automatic testing?
Does anyone know when this feature will be included?
Could someone confirm if I've now understood this correctly, that the MS APIs that support this* are still in preview? If so, I take it that is the blocker? And again if so, would someone happen to have any insights into when this might be released?
I understood that there are ppl working in collaboration with MS around this..?
*Managing EA Subscriptions through APIs
Hi all, could we have an update on this issue. Things looks to be in place to have something integrated in the official provider.
People at innovationnorway looks to have the code and the tests available and the hard limit have been raise to 2000.
Am I wrong or do we * just * need the PR and the review process ?
@rrey alas there's some upcoming changes coming on the Azure side that'll mean the existing API is unsuitable (it's being superseded) - we're working with the teams at Microsoft to unblock that - but we'll post more information when we have it.
@tombuildsstuff Could you elaborate what APIs will change? We're doing something similar as @joakimhellum describes (but then with an MCA based account, rather than a EA account), so I am curious if that will impact us.
@amochtar - I'll be working on this soon, and we're talking through the changes with MSFT at the moment, so it's not something we have pinned down right now. I'll link the work here when I start.
The schema for the new API is published: https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2020-09-01/Microsoft.Subscription.json
'aliases' is the resource type to create new subscriptions
The schema for the new API is published: https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2020-09-01/Microsoft.Subscription.json
'aliases' is the resource type to create new subscriptions
Thanks @matt-FFFFFF - I have that and chatting though implementation options with our MSFT Azure PM, slight delays due to timezone differences. Hoping to get started soon.
for azurerm_resource_group_template_deployment, resource will automatically attempt to delete resources deployed by the ARM Template when it is deleted
but not the same case for azurerm_subscription_template_deployment, can this be improved?
@theasphaltworld please open another issue to track that - since that request is unrelated to this issue
Hi
I can’t tell from the above comments: are there any potential chance this feature will be available for EA customers?
Thanks
Hi
I can’t tell from the above comments: are there any potential chance this feature will be available for EA customers?
Thanks
We have ongoing conversation with Microsoft on this, as soon as we have a clear path for what is and isn't supported we'll be creating this / these resources (we really want this feature too!)
Hi, any updates on this ? 🙂
Anyone looking for a way to do this while waiting, you can use this provider: https://registry.terraform.io/providers/innovationnorway/azure-preview/latest/docs/resources/azurepreview_subscription
I have been doing some work on this to create EA subscriptions programmatically, currently using an Azure Function with PowerShell. But have found the permissions/RBAC are the complex part of this, due to the new API for the billing account having a different set of roles & assignments than the rest of ARM.
e.g.
- ID: providers/Microsoft.Billing/billingAccounts/xxxxxxxx/enrollmentAccounts/xxxxxx/billingRoleDefinitions/c15c22c0-9faf-424c-9b7e-bd91c06a240b
- Role Name: Enrollment account owner
&
- ID: /providers/Microsoft.Billing/billingAccounts/xxxxxxxx/enrollmentAccounts/xxxxxx/billingRoleDefinitions/a0bcee42-bf30-4d1b-926a-48d21664ef71
- Name: Enrollment account subscription creator
There is no Terraform provider to enable the creation of these role assignments as it stands which will need adding to enable support.
@laingsc how did you handle the RBAC side of this for your provider, or are you just expecting this to be in place for the principal that runs the Terraform apply?
Happy to discuss further 😄
@laingsc you ideally need to create the sub outside of Terraform as, at the moment, the provider declaration is linked to a single subscription. Therefore, if you use TF to create the sub, then you'll need another plan created with a new provider declaration to be able to deploy resources to that subscription.
Hopefully this will change in the future :)
Ideally the azurerm provider would be completely decoupled from a subscription, and just simply be a login.... you can then have a generic "context" attribute which can be for an enrolment account, management group, subscription or resource group ... or even better / in addition, have providers able to be treated as just another node in the terraform resource dependency graph
@matt-FFFFFF I never though of the fact, that an instance of azurerm provider is actually a reference to a specific subscription, even though we use it regularly. I actually love the idea of @AdamCoulterOz with the generic context (enrolment account/tenant, subscription or even resource group) but I'm not sure if this is something that can be done, because this sounds to me like a major architectural design change.
@matt-FFFFFF ... azurerm v3? or maybe even a new provider (terraform-provider-az .... 😜) ?
@matt-FFFFFF @TheKangaroo - While a generic context is definitely desirable, Could you not just have the Azure provider instantiation as part of a module? Realistically, a lot of people are probably looking at this functionality for some form of landing zone functionality, so a module to declare what a subscription looks like seems pretty reasonable.
The end flow theoretically would look something like this?
- Central resources.tf would declare a subscription
- Output of the subscription resource would be provided to the module block
- Module will instantiate a new provider block, using the new subscription.
Unfortunately I don't readily have access to an EA account to test this with the aforementioned preview module. Happy to be corrected on my understanding of how Terraform providers work.
I don't think you can use values of an object and pass it to a provider for the run itself, but I never really tried that to be honest.
Right now we create subscriptions with a python script and use some folder structure magic and terragrunt for managing the subscription.
I think this could be way easier if we were able to have a provider for multiple resources.
This is not an absolut necessary feature for our use case, but I think this is necessary for this feature request since you cannot manage a subscription in terraform if the subscription itself is the boundary of a provider instance.
@jameswestall You cannot use generated values in a provider declaration. The providers are evaluated at init time so you must know the subscription ID at that point. You cannot do this if you are dynamically creating the subscription as part of the plan.
Hi All.
FWIW I have a branch in progress on this - but I've intentionally not pushed this at this time since it's not as usable as it first appears.
Whilst the API itself appears relatively simple, there's quite a bit to make it usable in Terraform/the Azure Provider due to the way this functionality works in Azure (primarily around usage and test coverage). On top of that, the API requires the calling user to have significant access to billing information, so I'm also looking into enabling the Billing provider to be able to access that safely. Be assured, we really want this too, so we're keeping things moving as much as we can.
Some additional info on the some of the limitations I mentioned:
-
There are a maximum number of Subscriptions that can be created against an Account. This is a maximum ever, so once it's reached that account can never have any more new subscriptions. For EA/Enrolment accounts this is 2000, but other account types it is as low as 20. (The subscriptions team are looking at increases here, but they're still talking in finite terms so the problem will still exist just take longer to hit)
-
Subscriptions cannot be deleted, only cancelled/deactivated. They are retained for up to 90 days for potential reactivation, then purged from the platform. They still exist in some backend systems, hence the previous lifetime limits above. Decoupling this is apparently being investigated.
-
A high number of subscriptions can have negative impact on performance of some operations, particularly in the portal.
-
Subscriptions created via the Alias API are always created in the Home Tenant of the user account that submitted the request, this may not be the desired location of the subscription, or even the tenant on which Terraform is operating when it calls the API, we're looking at what can be done here. (Side note - it appears that it must be a user account to create a subscription; MSI and SP do not appear to be supported, which obviously presents issues for most CI approaches.) There are APIs for moving Subscriptions between Tenants, I'm waiting on details from the Subscriptions team at Azure on these.
The broader discussion around the provider/client auth is tangentially related but off-topic - to keep this issue focused on the original issue, I'm going to mark those comments as off-topic for the moment - however should you wish to discuss this further I'd point you towards the community discuss forum.
Whilst we appreciate it's frustrating this isn't available natively at this time, we are working closely with the teams at Microsoft to unblock this in the background - but there's still a number of blockers for this which we need to solve before we can do so - we'll post an update once we have more information.
Thanks!
@jackofallops We now support SPNs with new API:
https://github.com/Azure/Enterprise-Scale/blob/main/docs/Deploy/enable-subscription-creation.md
@jackofallops They also support MSIs as well. Checkout this project I created where I use an Azure Function to create the subs with an MSI. https://github.com/jtracey93/AzureSubscriptionVendingFunction
This has been released in version 2.50.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:
provider "azurerm" {
version = "~> 2.50.0"
}
# ... other configuration ...I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!