[Bug] Cannot apply Terraform when appending a new certificate to accepted_client_ca while the change has been made
Closed this issue · 2 comments
What are you really trying to do?
I am testing rotating certificates by appending a new one to accepted_client_ca.
For context, I created a Terraform module to integrate with secretmanager where we store the certificates.
We have a job that rotates the certificates by appending to the secretmanager and we expect to do a terraform apply after the secret in secretmanager is updated. We want to hide these details so we created Terraform module which basically wrap the temporalcloud_namespace with aws_secretsmanager_secret.
data "aws_secretsmanager_secret" "certs" {
name="${var.secret_name_prefix}/${var.environment}/${var.namespace}/certs"
}
data "aws_secretsmanager_secret_version" "certs" {
secret_id=data.aws_secretsmanager_secret.certs.id
}
resource "temporalcloud_namespace" "temporal_namespace" {
name=var.namespace
regions=[var.region]
accepted_client_ca=base64encode(data.aws_secretsmanager_secret_version.certs.secret_string)
retention_days=var.retention_days
}
This is how I define the namespace:
module "cloud-ops-temporal-namespace" {
source="./modules/temporal-namespace"
namespace="cloud-ops"
environment="prod"
region="aws-us-east-1"
retention_days=7
}
Describe the bug
After I set the initial certificates, I successfully set up my namespace using Terraform.
However, after I add a new certificate and apply again, there was an error:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace will be updated in-place
~ resource "temporalcloud_namespace" "temporal_namespace" {
~ accepted_client_ca = (sensitive value)
~ endpoints = {
~ grpc_address = "cloud-ops.xxxx.tmprl.cloud:7233" -> (known after apply)
~ web_address = "https://cloud-ops.xxxxxxx.web.tmprl.cloud" -> (known after apply)
} -> (known after apply)
id = "cloud-ops.xxxxxxx"
name = "cloud-ops"
# (2 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace: Modifying... [id=cloud-ops.xxxxxxx]
module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace: Still modifying... [id=cloud-ops.xxxxxxx, 10s elapsed]
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace, provider "provider[\"registry.terraform.io/temporalio/temporalcloud\"]" produced
│ an unexpected new value: .accepted_client_ca: inconsistent values for sensitive attribute.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
When I tried to apply again, it's a different error:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace will be updated in-place
~ resource "temporalcloud_namespace" "temporal_namespace" {
~ accepted_client_ca = (sensitive value)
~ endpoints = {
~ grpc_address = "cloud-ops.xxxxxxx.tmprl.cloud:7233" -> (known after apply)
~ web_address = "https://cloud-ops.xxxxxxx.web.tmprl.cloud" -> (known after apply)
} -> (known after apply)
id = "cloud-ops.xxxxxxx"
name = "cloud-ops"
# (2 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace: Modifying... [id=cloud-ops.xxxxxxx]
╷
│ Error: Failed to update namespace
│
│ with module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace,
│ on modules/temporal-namespace/main.tf line 17, in resource "temporalcloud_namespace" "temporal_namespace":
│ 17: resource "temporalcloud_namespace" "temporal_namespace" {
│
│ rpc error: code = InvalidArgument desc = nothing to change
╵
I went to the console to check my CA certificate of my namespace. There are actually two.
But right now I cannot apply the Terraform and there is always a drift.
# module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace will be updated in-place
~ resource "temporalcloud_namespace" "temporal_namespace" {
~ accepted_client_ca = (sensitive value)
~ endpoints = {
~ grpc_address = "cloud-ops.xxxxxxx.tmprl.cloud:7233" -> (known after apply)
~ web_address = "https://cloud-ops.xxxxxxx.web.tmprl.cloud" -> (known after apply)
} -> (known after apply)
id = "cloud-ops.xxxxxxx"
name = "cloud-ops"
# (2 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
Minimal Reproduction
Environment/Versions
- OS and processor: MacBook Pro (M2 Max)
- Temporal Version: N/A
- Are you using Docker or Kubernetes or building Temporal from source? N/A
Terraform information:
Terraform v1.5.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.50.0
+ provider registry.terraform.io/temporalio/temporalcloud v0.0.8
TL;DR: It is very sensitive with \n and there are slightly different behaviors when adding new certificate vs removing old certificate. The solution ended up being making sure there is no extra '\n' between certificates and and there is a newline by the end the secret (certificates).
It turned out to be the reason that I added an extra \n between certificates. Now it works well for the case of appending new certificate.
However I got a similar error when removing the first certificate.
"""
-----BEGIN CERTIFICATE-----
1st
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
2nd
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
3rd
-----END CERTIFICATE-----
=>
-----BEGIN CERTIFICATE-----
2nd
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
3rd
-----END CERTIFICATE-----
"""
This is the error I got:
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to module.cloud-ops-temporal-namespace.temporalcloud_namespace.temporal_namespace, provider "provider[\"registry.terraform.io/temporalio/temporalcloud\"]" produced
│ an unexpected new value: .accepted_client_ca: inconsistent values for sensitive attribute.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
I look into the differences between the certificates saved in the secretmanager and showing in the Temporal Cloud UI.
The only difference is that in Temporal Cloud UI the certificates has a new line by the end
I tested it by manually adding \n to the secret in secretmanager and the terraform plan now says
No changes. Your infrastructure matches the configuration.
Thank you for the report! A fix for this has shipped in v0.0.10.