container_definitions when using dynamic variables is failing when nothing is deployed yet

Description

When using dynamic variables (e.g. outputs from other modules like terraform-aws-rds) in container_definitions, the Terraform fails when aforementioned resource does not exist yet.

✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]: 5.2.2
Terraform version:

Terraform v1.6.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.23.1
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/ns1-terraform/ns1 v2.0.10

Reproduction Code [Required]

Full setup tree (you'll need AWS and NS1 credentials, however you don't have to have valid one to get a relevant error, just export NS1_APIKEY=xxx): https://github.com/ignatenkobrain/terraform-netbox-aws/tree/ac1a97ac1b11694f16fffa1cdcc3e3e5d6e2a772

Steps to reproduce the behavior:

export AWS_...
export NS1_APIKEY=secret
terraform init
terraform apply

Expected behavior

No error message and all resources are being properly created.

Actual behavior

Plan: 54 to add, 0 to change, 0 to destroy.
╷
│ Error: Invalid for_each argument
│ 
│   on .terraform/modules/app/modules/service/main.tf line 526, in module "container_definition":
│  526:   for_each = { for k, v in var.container_definitions : k => v if local.create_task_definition }
│     ├────────────────
│     │ local.create_task_definition is true
│     │ var.container_definitions will be known only after apply
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵

Terminal Output Screenshot(s)

Additional context

If I change

terraform-aws-ecs/main.tf

Line 102 in e987d95

container_definitions = try(each.value.container_definitions, {})

line to not use try(each.value.container_definitions, {}) to simply each.value.container_definitions, it starts to work without any issues.

I would suggest using the service module separate from the cluster module, like this example

terraform-aws-ecs/examples/fargate/main.tf

Line 55 in e987d95

module "ecs_service" {

@bryantbiggs This does solve the issue, thank you! Would you like to keep this ticket open for any actual improvements?

Let me think on it, so lets keep it open. In theory, it would be nice if this single module worked the same as the composition of the cluster+service modules, but sometimes iterating over maps of maps with computed values there are "issues" like this

Yeah the strangest thing about this issue is that it's sporadic- sometimes an apply will just work. Definitely has been an issue for a while now. Targeting worked for me in the past.

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

Any news on this? I have got the samething.
on .terraform/modules/ecs.ecs/modules/service/main.tf line 525, in module "container_definition":
│ 525: for_each = { for k, v in var.container_definitions : k => v if local.create_task_definition && try(v.create, true) }
│ ├────────────────
│ │ local.create_task_definition is true
│ │ var.container_definitions will be known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the
│ full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results
│ only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a
│ second time to fully converge.

~~It's definitely annoying. I've been using -target as suggested to avoid this collision, which seems really sporadic.~~

target doesn't work for us long term. I recommend using what OP did in the additional context section of this issue. If you are early on in the process, I recommend not using this module for a full ECS deploy altogether.

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

This issue was automatically closed because of stale in 10 days

It's definitely annoying. I've been using -target as suggested to avoid this collision, which seems really sporadic.

Hi iycbell, I have a similar issue. How do I use this -target option? I mean, do I need to pass something as a target?

Btw, I am using cdktf plan.

It's definitely annoying. I've been using -target as suggested to avoid this collision, which seems really sporadic.

Hi iycbell, I have a similar issue. How do I use this -target option? I mean, do I need to pass something as a target?

Btw, I am using cdktf plan.

Hi donatoaz. This was a weird issue, but ultimately target was a red herring. The plans eventually failed again. A more permanent fix for us was following what OP did in the additional context section:
Updating this line in the module to each.value.container_definitions. The logic is that if there is a failure to set the container definitions, it should be caught during our plan/apply. Good luck.

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.