Feature Request: Splat for all resources of a type
philomory opened this issue · 27 comments
Current Terraform Version
Terraform v0.12.0-alpha4 (2c36829d3265661d8edbd5014de8090ea7e2a076)
+ provider.random v2.0.0-5-g612dff2-dev
Use-cases
I'd like to be able to use resource_type.*.id to get a list of the id of every resource of a type; currently, you can only do this for resources that have a count attribute (or eventually for_each, I guess). Sometimes, you have a bunch of resources you want to declare explicitly because they don't necessarily have very much in common, but you still want to refer to them all from another resource.
My explicit use-case is the third-party auth0 provider. I've got a bunch of auth0_client resources that all have their own custom configuration. Then, I've got an auth0_connection resource which represents our actual identity provider (an LDAP server). In creating the auth0_connection resource, I need to pass a list of auth0_client ids that that connection will be used for. And I'd love to get that list of ids by simply writing auth0_client.*.client_id. But that's not possible.
Attempted Solutions
This didn't work:
resource "random_id" "foo" {
byte_length = 1
}
resource "random_id" "bar" {
byte_length = 1
}
output "example" {
value = "${random_id.*.hex}"
}
I got this error:
$terraform validate
Error: Invalid reference
on test.tf line 12, in output "example":
12: value = "${random_id.*.hex}"
A reference to a resource type must be followed by at least one attribute
access, specifying the resource name.
A similar attempt with random_id[*].hex also failed.
I have a need for the same use case as described above.
In addition, I'd like this 'resource type splat' feature to be usable in conjunction with the feature described in issue #9067 (and/or the upcoming for/for_each with optional filtering via an if qualifier) to enable me to create a single output variable whose value is a list of some or all of the resources (entire resources, not just IDs, or names) of a given type, without specifying or knowing the names or IDs of the included resources.
The need for this is based on my use of code generation tools to generate HCL dynamically. I want separate parts of my code generation to be able to generate HCL that will be combined with each other and applied as a single set of TF configuration, but also to have my separate code generation components to remain encapsulated and unaware of each other. But I'd like to have 1 component that generates output variables that collect all or some of the resources (and/or specific computed/exported attributes of them) into named output variables.
For example:
output "all_eips_list" {
value = "${aws_eip.*}"
}
output "all_eip_public_ips_list" {
value = "${aws_eip.*.public_ip}"
}
output "all_eip_public_to_private_ips_map" {
# Result is a map from eip public IP to private IP address, such as:
# {"192.168.1.2" = "10.1.2.3", "192.168.1.5" = "10.2.3.4"}
value = {
for eip in aws_eip.*:
eip.public_ip => eip.private_ip
}
}
output "some_eip_public_to_private_ips_map" {
# Result is a map from eip public IP to private IP address, such as:
# {"192.168.1.2" = "10.1.2.3", "192.168.1.5" = "10.2.3.4"}
value = {
for eip in aws_eip.*:
eip.public_ip => eip.private_ip
if eip.associate_with_private_ip_address
}
}
Similar use cases exist for aws_lb.dns_name, aws_s3_bucket, and many other resources where the HCL configuration may contain a dynamic number of a specific type of resource (but not count-based repetition; each resource of the same type is explicitly declared separately, distinctly, and independently, as each resource of the same type has unique properties and configuration) and computed or exported attributes that aren't known until terraform apply or terraform refresh is executed.
A further (related but distinct) use case is that output variables similar to the examples above would (in many cases) be useful as 'reusable' HCL code, and could be included (without requiring any changes to the output variable HCL nor any configuration via input variables, etc.) in almost any arbitrary TF configuration that includes the referenced providers and resource types.
This would be useful to people that manage multiple TF configurations as a way of providing standardized (i.e. consistent across TF configurations, but with custom content and format determined by the output variables' reusable HCL definition) outputs.
This 'reusable outputs' use case applies equally well regardless of whether the separately managed TF configurations are manually generated or generated by code or by some other means.
Also, given that this change probably requires a change to HCL itself, it'd be great if this could be included in the initial v0.12 release. Is there still a chance of that?
One more use case:
I am creating N identical dynamodb tables in a module.
Module is called for each region (regional module).
In a root module, I need to create global dynamodb tables and I need regional table names.
It would be good, to return this output from a regional module:
output "table_names" {
value = aws_dynamodb_table.*.name
}
yes need this for output vars - different customized vms (windows and linux) created by module as different resources the same type!
I was recently working on getting some of the less common resources tagged and some kind of ability like this would have been really useful to avoid needing to maintain a list of resources:
locals {
resources_tags = setproduct(
concat(
values(aws_efs_mount_target.a)[*].network_interface_id,
values(aws_efs_mount_target.b)[*].network_interface_id,
values(aws_efs_mount_target.c)[*].network_interface_id,
values(aws_efs_mount_target.d)[*].network_interface_id,
values(aws_efs_mount_target.e)[*].network_interface_id,
module.vpc.vpc_endpoint_efs_network_interface_ids,
module.vpc.private_route_table_ids,
module.vpc.public_route_table_ids,
),
[for key, value in local.tags :
[key, value]
]
)
}
resource "aws_ec2_tag" "all" {
count = length(local.resources_tags)
resource_id = local.resources_tags[count.index][0]
key = local.resources_tags[count.index][1][0]
value = local.resources_tags[count.index][1][1]
}Here we are, approaching 3 years after this was opened. Is this at all likely to happen?
I need this to for_each for github_repositories too.
For now, I've quickly worked around it using an external program to generate the list from the terraform state for any given resource type. The terraform_resources.sh script can be found here:
https://github.com/HariSekhon/DevOps-Bash-tools
and working Terraform code using it can be found here:
https://github.com/HariSekhon/Terraform
in the github_team_repository.tf and github_branch.tf files - each of which are getting different attributes using that script.
Jumping on this bandwagon, for called modules within a workspace. I need to output certain metadata values for each system built by a module, and I can't use foreach as each system needs to be able to have different providers specified for its module call.
Example: This is currently the only way I got it working. This is NOT manageable at scale at all. Each new server built requires the sys admins to also add to the output value... which people have already forgotten, causing failed builds.
module "srv-1" {
providers = {...}
}
module "srv-2" {
providers = {... something different ...}
}
output "all-systems" {
value = merge(module.srv-1, module.srv-2)
}
All we really need is the ability for "outputs" to iterate over the built resources and modules in state, instead of needing to know the exact resource id ahead of time.
Adding my use case:
Currently creating a bunch of aws_apigatewayv2_domain_name's to be later used in aws_apigatewayv2_api_mappings
Would be nice to have something similar to the following:
resource "aws_apigatewayv2_domain_name" "www"{
domain_name = var.env == "prod" ? "www.${var.domain}.co.uk" : "www.${var.env}.${var.domain}.co.uk"
}
resource "aws_apigatewayv2_domain_name" "top"{
domain_name = var.env == "prod" ? "${var.domain}.co.uk" : "${var.env}.${var.domain}.co.uk"
}
then a local similar to this:
locals {
domains = [aws_apigatewayv2_domain_name.*.domain_name]
}
to then be referenced with:
resource "aws_apigatewayv2_api_mapping" "domain" {
for_each = for_each(local.domains, "domain")
....
}
A use case in Azure: creating diagnostics settings for a bunch of resources of a type
resource "azurerm_monitor_diagnostic_setting" "keyvaults" {
for_each = toset(azurerm_key_vault.*.id) # <-- or something like that
name = "storage-export"
target_resource_id = each.key
storage_account_id = azurerm_storage_account.diag_logs.id
log {
category = "AuditEvent"
enabled = true
retention_policy {
enabled = true
days = 0
}
}
log {
category = "AzurePolicyEvaluationDetails"
enabled = true
retention_policy {
enabled = true
days = 0
}
}
}Piling on the use case list, this would be great for setting dynamic resource dependencies when you want a resource to depend on all resources of another type:
resource "type" "name" {
depends_on = trigger_resource_type.*
...
}Useful for working with the Okta provider, specifically when using okta_group_role resource.
resource "okta_group" "admins-group" {
name = "admins"
description = "Admins"
}
resource "okta_group" "reader-group" {
name = "readers"
description = "Readers"
}
resource "okta_group_role" "admin-group-role" {
group_id = okta_group.admins-group.id
role_type = "USER_ADMIN"
target_group_list = okta_group.*.id
}
Useful for working with the Okta provider, specifically when using
okta_group_roleresource.resource "okta_group" "admins-group" { name = "admins" description = "Admins" } resource "okta_group" "reader-group" { name = "readers" description = "Readers" } resource "okta_group_role" "admin-group-role" { group_id = okta_group.admins-group.id role_type = "USER_ADMIN" target_group_list = okta_group.*.id }
A similar problem led me here too. What I ended up doing was putting the group constituents into YAML files so I could then use fileset to consolidate in more creative ways in the terraform files.
Useful for working with the Okta provider, specifically when using
okta_group_roleresource.resource "okta_group" "admins-group" { name = "admins" description = "Admins" } resource "okta_group" "reader-group" { name = "readers" description = "Readers" } resource "okta_group_role" "admin-group-role" { group_id = okta_group.admins-group.id role_type = "USER_ADMIN" target_group_list = okta_group.*.id }A similar problem led me here too. What I ended up doing was putting the group constituents into YAML files so I could then use fileset to consolidate in more creative ways in the terraform files.
Can you expand on this? I will likely need to end up doing similar as a workaround until this feature has been implemented.
Did the files just contain the individual groups? How did you reference the files when building your list for okta_group_role? I am not familiar with fileset.
Mine is for Azure DevOps and I specifically wanted a file per set of groups because of how our code review process works, but I also wanted a distinct list of all users in all defined groups. The idea should allow more creative groups of things in general though. I made a locals that looks something like:
locals {
groups = flatten([
for group_file in fileset(path.module, "groups/*.yml") : yamldecode(file("${path.module}/${group_file}"))["groups"]
])
data_users = distinct(flatten([for group in local.groups : lookup(group.members, "users", [])]))
data_groups = distinct(flatten([for group in local.groups : lookup(group.members, "groups", [])]))
}
data "azuredevops_group" "groups" {
project_id = data.azuredevops_project.project.project_id
for_each = toset(local.data_groups)
name = each.value
}
To add another use case on a (almost) 3 year old issue, I am trying to create a cloudwatch dashboard in aws with the alarm widget. The cloudwatch widget only accepts arns. Would be super useful if I could make my code look something like this:
resource "aws_cloudwatch_dashboard" "Alarm-Dashboard" {
dashboard_name = "Alarm-Dashboard"
dashboard_body = <<EOF
{
"widgets": [
{
"type": "alarm",
"x": 0,
"y": 0,
"width": 24,
"height": 11,
"properties": {
"title": "",
"alarms": "[${aws_cloudwatch_metric_alarm.*.id}]"
}
}
]
}
EOF
}
Can we get insights from the developers if this is worked on or why this is being withheld?
@mBlomsterberg It is not being worked on. It is currently at # 33 of the most-upvoted issues. We use upvotes as a method to help feed our product backlog. Thanks for your question!
Got another use case for making it easier to access lambdas in a template file
resource "aws_api_gateway_rest_api" "example" {
name = "example"
description = "example"
body = templatefile("${file("../backend/api.yaml")}", {
//gives access to lambda[func_name] inside api.yml
lambdas = { for lambda in aws_lambda_function : lambda.handler => lambda.invoke_arn }
})
}
Any progress - would be very handy!
Any progress - would be very handy!
Here's a workaround I found:
resource "aws_lambda_function" "test" {
}
//Dont forget to add this
//once this issue is passed https://github.com/hashicorp/terraform/issues/19931#
locals {
lambdas = [
aws_lambda_function.test
]
}
resource "aws_api_gateway_rest_api" "example_api" {
body = templatefile("../backend/api.yaml",
{ for lambda in local.lambdas : lambda.handler => lambda.invoke_arn }
)
}
just adding all the resources you need (addressed by name)
to an array, it's a little annoying, but I've found it gets the job done for now, it should let you splat the array and therefore splat the resources inside 🤷
I thought this was a very basic syntax that should be supported
Getting the list of the same resource type is very useful
would love this feature
Adding another +1 on this feature would be great to have.
This is something our organization really needs. Please prioritize this as it also seems that others have need for this feature as well 🚀
For sure this is exactly what I would like to have so that I could do something like this for dynamically defined outputs:
resource "coralogix_webhook" "opsgenie" {
for_each = local.opsgenie_team_keys
name = "opsgenie-${each.key}"
opsgenie = {
url = "https://api.opsgenie.com/v1/json/?apiKey=${each.value.api_key}"
}
}
resource "coralogix_webhook" "jira" {
for_each = local.jira_team_keys
name = "jira-${each.key}"
jira = {
url = "https://redacted.atlassian.net"
project_key = each.value.project_key
}
}
#
# Then with this type of output we would be able to define the following
#
output "coralogix_webhooks" {
value = { for hook in coralogix_webhook.* : hook.name => hook.id }
}