error when default_tags are identical to resource tags

I am confused why this condition causes an error.

Error: "tags" are identical to those in the "default_tags" configuration block of the provider: please de-duplicate and try again

terraform-provider-aws/aws/tags.go

Line 108 in ac06ced

if defaultTagsConfig.TagsEqual(resourceTags) {

We have many situations where a useful tag value can be set in the provider default_tags and then be overridden in the majority of resources, but be explicitly the same in a small subset. These situations are generally in modules rather than top level resources, so it is very difficult to selectively remove identical tag key/value pairs.

If the default tag values are overridden by resource tags anyway why does it matter if they are identical?

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.7

provider registry.terraform.io/hashicorp/aws v3.38.0

Affected Resource(s)

all

aws_XXXXX

Related: #7926.

Hi @devopsrick, thank you raising this issue. As you found in the code, the error message was by design as having identical provider-level and resource-level tags definitions proves difficult to differentiate when the data is returned from an AWS API (since the provider/resource-level tags concept is only known within the provider, not an AWS service) and thus we cannot know with certainty if the data is to be stored in the resource's tags argument (and if data is incorrectly stored into a resource's tags, would result in a perpetual diff), or only into the tags_all Computed argument that holds all known tags for the resource; to appropriately differentiate, we would need to read from resource's state with d.Get("tags") to determine if the tags were originally configured there, but this is generally discouraged across provider resources during the Read CRUD operation. As this feature is new to the provider, however, we are working on evaluating what our user's may need to make default_tags adoption a smooth process, and the intended behavior presented here brings to light an additional edge case that may become prevalent with more users and we welcome more feedback related to this error messaging. As a start, we can improve the documentation to note this behavior 👍

For additional context for the reason for the error messaging/preventing a perpetual diff, we can see here in the VPC resource that when we set the resource tags API object in state, we need to remove any tags configured at the provider-level with the method RemoveDefaultConfig. So in the case of identical provider-level and resource-level tags configurations, the result of tags.RemoveDefaultConfig(defaultTagsConfig) would be an empty map without prior knowledge as noted above.

https://github.com/hashicorp/terraform-provider-aws/blob/main/aws/resource_aws_vpc.go#L318-L323

One thing we don't guard against, and important to note as well, is duplicate key/value pairs defined in the provider and resource tags. In the example below, this will apply without error but a perpetual diff will appear, such that terraform suggests an update-in-place for the duplicate tag Name = "Example" since the tag gets removed as mentioned above with RemoveDefaultConfig.

provider "aws" {
  default_tags {
     tags = {
      Name = "Example"
    }
  }
}

resource "aws_vpc" "example" {
  tags = {
    Name = "Example"
    Owner = "Other"
  }
}

Ok I see the distinction now. Thank you!

@anGie44 So it's kind of by design that applying the same tags with the same values on provider and on resource level triggers a perpetual diff?
We've hit this issue using our Company's TF modules which apply tagging to some but not all AWS resources and these modules are used by lots of teams hence we can't just switch from applying tags on resource level in TF modules to applying tags on provider level in TF callers. Is there a way for us to have resource and provider level tags with the same name/value in the same time?

Can we re-open this issue for discussion/tracking? While the explanation makes sense, this does really limit the usefulness of this new feature. Specifically for a company with a large module registry and configuration templates that currently define the org mandated default tags within locals, taking advantage of provider level tags would be nearly impossible unless they can be overwritten at the resource level.

Is there a way for us to have resource and provider level tags with the same name/value in the same time?

Hi @yermulnik 👋 , at the moment, I'm not aware of a workaround that allows for this without ending in a cycle of perpetual diffs. But as per @scarbeau's suggestion, I'm happy to reopen this for more discussion/comments related to how this behavior may be impacting the usefulness of the feature.

taking advantage of provider level tags would be nearly impossible unless they can be overwritten at the resource level.

Hi @scarbeau , thank you for commenting here! Just wanted to double-check with your statement above, do you mean "overwritten" in the sense that you cannot exclude a provider-level tag from a resource but only "overwrite" the tag key's value?

@anGie44 I'd imagine resource level tags to be merged on top of provider level tags so that those which are the same get overwritten by resource level tags w/o having a perpetual diff. This will definitely ease transition from resource level tags to provider level tags w/o a need to modify existing resource level tags along with an option of having mandatory tags forcibly applied on resource level even if these are supplied by TF module caller via AWS provider configuration block.

Thanks for re-opening this issue for further discussion.

oy, I wish I had known this before applying the default tags

Oof, just hit this, trying to use the feature for the first time. The default_tags feature is nearly useless right now, between the outright error and the perpetual diff.

I second the idea that resource-level tags should be merged and override tag keys from default_tags.

I'm hitting the "perpetual diff" problem too, making "default_tags" difficult to use effectively.

I also agree that the resource-level tags should override tag keys from default_tags.

CDK has some interesting ideas about merging / overriding tags: https://docs.aws.amazon.com/cdk/latest/guide/tagging.html

I found a workaround which can be used in modules where default_tags may or may not be set on the AWS provider.

In the module, fetch the default tags using a data source and then set a local variable for the tags:

data "aws_default_tags" "default_tags" {}

locals {
  # Set common tags unless already defined in aws default tags
  common_tags = length(data.aws_default_tags.default_tags.tags) > 0 ? {} : {
    Environment = var.environment
    Service     = var.service
  }
}

So we set common_tags to an empty hash if default tags exist, so the default tags will be applied. If there are no default_tags, then we set them here in common_tags.

To use in a resource, just set the tags to local.common_tags, and we can also merge in additional tags:

resource "aws_alb_target_group" "alb_target_group" {
  ...

  tags = merge(
    local.common_tags,
    {
      Name = var.name
    }
  )
}

I tested this with and without default_tags on the provider. In both cases the desired tags are set and there is no perpetual diff problem.

The above workaround looked very promising but I ended up with
Error: Cycle: data.aws_default_tags.default_tags, local.common_tags (expand), provider["registry.terraform.io/hashicorp/aws"]
error. That was with AWS provider 3.52.0; stopped using default_tags in the provider block just for now.

As a side note, if any AWS engineers are watching this issue, it would be nice if the cloud itself supported better methods of organizing tags. It would be nice to be able to create a tagged "container" and associate arbitrary AWS resources with that "container" and have those resource automatically inherit the tags. Managing tags in AWS is still a complete nuisance, even with tools like Terraform or CDK or whatever

Here's another work-around to avoid adding identical tags at the resource level if they are defined at the provider level:

data "aws_default_tags" "provider" {}

locals {
  tags = {
    for k, v in {
      env   = var.environment
      stack = var.stack
    } : k => v if lookup(data.aws_default_tags.provider.tags, k, null) == null || lookup(data.aws_default_tags.provider.tags, k, null) != v
  }
}

resource "aws_route53_zone" "zone" {
  name    = var.domain_name
  comment = "Zone for ${var.environment} environment."

  vpc {
    vpc_id = var.vpc_id
  }

  tags = local.tags
}

Edit: the use of the aws_default_tags datasource requires terraform aws provider v3.43.0 or newer, see #19391

I find the error text baffling...

Error: "tags" are identical to those in the "default_tags" configuration block of the provider:

I parse that as saying all the tags in "tags" are identical to all the tags in "default_tags". Maybe some confusion can be avoided by changing the error text. I know it would have helped me!

Error: Some key/value pairs in "tags" are identical to those in the "default_tags" configuration block of the provider:

How about x and y tag overlap or x tags must be distinct from y?

would there be any resolution of this in future? Any timelines?

Any update on this? Would be good to get a fix as it's impacting downstream modules as well

This was resolved in #30793 and merged to main in #31392. Feature will be released in v5.0.0

This functionality has been released in v5.0.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.