terraform-google-modules/terraform-google-pubsub

Can not use dynamic service account

Closed this issue · 5 comments

Tolsee commented

TL;DR

Cannot use dynamic value of service account in service_account of pull_configuration. It needs to be static.

Expected behavior

Allow the use dynamic value on service_account of pull_configuration.

Observed behavior

Got following error:

341: for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "service_account", null) != null } : {}

  var.create_subscriptions is true
  var.pull_subscriptions is list of map of string with 1 element

The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.

When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.

Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

Terraform Configuration

module "using_this module" {
  // ...
  pull_subscriptions = [
    {
      name                 = var.some_name
      service_account      = google_service_account.default
    }
  ]
}

Terraform Version

1.3.6

Additional information

I was think we could introduce another key like create_iam

module "using_this module" {
  // ...
  pull_subscriptions = [
    {
      name                 = var.some_name
      create_iam           = true
      service_account      = google_service_account.default
    }
  ]
}

And update like this:
Current:

terraform-google-pubsub/main.tf

Lines 340 to 362 in 3670eaf

resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_subscriber" {
for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "service_account", null) != null } : {}
project = var.project_id
subscription = each.value.name
role = "roles/pubsub.subscriber"
member = "serviceAccount:${each.value.service_account}"
depends_on = [
google_pubsub_subscription.pull_subscriptions,
]
}
resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_viewer" {
for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "service_account", null) != null } : {}
project = var.project_id
subscription = each.value.name
role = "roles/pubsub.viewer"
member = "serviceAccount:${each.value.service_account}"
depends_on = [
google_pubsub_subscription.pull_subscriptions,
]
}

Proposed:

resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_subscriber" {
  for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "create_iam", false) != false } : {}

  project      = var.project_id
  subscription = each.value.name
  role         = "roles/pubsub.subscriber"
  member       = "serviceAccount:${each.value.service_account}"
  depends_on = [
    google_pubsub_subscription.pull_subscriptions,
  ]
}

resource "google_pubsub_subscription_iam_member" "pull_subscription_sa_binding_viewer" {
  for_each = var.create_subscriptions ? { for i in var.pull_subscriptions : i.name => i if lookup(i, "create_iam", false) != false } : {}

  project      = var.project_id
  subscription = each.value.name
  role         = "roles/pubsub.viewer"
  member       = "serviceAccount:${each.value.service_account}"
  depends_on = [
    google_pubsub_subscription.pull_subscriptions,
  ]
}

If this is okay, I can work on the PR. Also, happy to learn if you have other suggestions.

github-actions commented

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

scipe commented

Hey friends! Is any updates on it? Hit exact the same issue. I did workaround inside my Terraform, but will be nice to get fresh version of pubsub module.

  • 👍4
aarozhkov commented

We should reopen this issue.
IMHO this is standard case when you create a service account and try to attach it to pub-sub in a single module/state.
There should be solution for that.

islamsamipkr commented

Hi guys, got hit with the same issue, we are trying to assign all the writer identities generated from gcp pubsub iam binding to publisher from one project to another, we are getting intermittent issues. It gets applied and the iam pubsub binding works, but then when a new writer identity needs to get assigned the permission, all the other service account gets changed automatically without any change whatsoever, which terraform version should we use in this case? does it need to be version locked?
image

Reproduction steps:
module:

module "app-pubsub" {
  source   = "terraform-google-modules/pubsub/google"
  version  = "~> 5.0"
  for_each = { for x, n in var.pubsub_config : x => n }
  project_id = var.project_id
  grant_token_creator = false
  topic               = each.value.pubsub_topic_name
  topic_labels        = each.value.pubsub_topic_labels
  subscription_labels = each.value.subscription_labels

  pull_subscriptions = [
    {

      name = each.value.pubsub_subscription_name

      message_retention_duration = "604800s" // Default (7 days)
      retain_acked_messages      = false     // Default
      ack_deadline_seconds       = 10        // Default
      expiration_policy          = ""        // Subscription never expires
      dead_letter_topic     = "projects/${var.project_id}/topics/${each.value.pubsub_subscription_name}_dl"
      max_delivery_attempts = 5      // Default
      minimum_backoff       = "10s"  // Default
      maximum_backoff       = "600s" // Default

      enable_message_ordering      = true
      enable_exactly_once_delivery = true

      service_account = var.pubsub_subscription_sa
    }
  ]

  depends_on = [
    module.pubsub_dl
  ]
}`
 

`resource "google_logging_project_sink" "app_project_sink" {
  for_each    = { for x, n in var.project_sink_config : x => n }
  name        = "${each.value.project_name}-logsink-${index(var.project_sink_config, each.value) + 1}"
  description = "Project sink for ${each.value.project_name}"
  destination = "pubsub.googleapis.com/projects/${var.project_id}/topics/${each.value.pubsub_topic_name}"

  filter                 = each.value.filter
  project                = each.value.project_id
  unique_writer_identity = true

}


resource "google_pubsub_topic_iam_binding" "project_sink_writer" {
  for_each = { for x, n in google_logging_project_sink.app_project_sink : x => n }
  project  = var.project_id
  topic    = element(split("/", each.value.destination), length(split("/", each.value.destination)) - 1)
  role     = "roles/pubsub.publisher"
  members = [
    each.value.writer_identity
  ]`
```
![image](https://github.com/terraform-google-modules/terraform-google-pubsub/assets/11166810/42a54bf6-cf15-4a2f-aa99-8c558fd00aa6)
github-actions commented

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days