use existing k8s sa fails
gvasquez-plenty opened this issue · 8 comments
gvasquez-plenty commented
Hello,
I'm trying to annotate an existing k8s sa but I'm getting this:
Error: local-exec provisioner error
│
│ with module.annotate-sa.module.gcloud_kubectl.null_resource.run_destroy_command[0],
│ on .terraform/modules/annotate-sa/main.tf line 258, in resource "null_resource" "run_destroy_command":
│ 258: provisioner "local-exec" {
│
│ Error running command 'PATH=/google-cloud-sdk/bin:$PATH
│ .terraform/modules/annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh
│ dev-cluster us-central1 false false kubectl annotate sa -n kube-system
│ external-dns iam.gke.io/gcp-service-account-
Do you guys knows how to fix this?
terraform version : 1.0.0
google: 3.53
kubernetes: 2.3.2
bharathkkb commented
Hi @gvasquez-plenty
Could you paste the full error log and an example configuration for reproduction?
gvasquez-plenty commented
Hello @bharathkkb
This is the example configuration for reproduction, I'm using terragrunt:
# These are the variables we have to pass in to use the module specified in the terragrunt configuration above
inputs = {
name = "external-dns"
project_id = local.project
location = local.region
cluster_name = dependency.gke-cluster.outputs.name
namespace = "kube-system"
roles = ["roles/dns.admin"]
use_existing_k8s_sa = true
}
and this is the output error:
module.annotate-sa.module.gcloud_kubectl.null_resource.additional_components_destroy[0]: Creating...
module.annotate-sa.module.gcloud_kubectl.null_resource.additional_components_destroy[0]: Creation complete after 0s [id=6871260258749315415]
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0]: Creating...
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0]: Provisioning with 'local-exec'...
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Executing: ["/bin/sh" "-c" "PATH=/google-cloud-sdk/bin:$PATH\n.terraform/modules/annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh dev-cluster us-central1 false false kubectl annotate --overwrite sa -n kube-system external-dns iam.gke.io/gcp-service-account=external-dns@development-319813.iam.gserviceaccount.com\n"]
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + '[' 12 -lt 5 ']'
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + CLUSTER_NAME=dev-cluster
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + LOCATION=us-central1
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + PROJECT_ID=false
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + INTERNAL=false
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + USE_EXISTING_CONTEXT=kubectl
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + ENABLE_IMPERSONATE_SERVICE_ACCOUNT=annotate
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + IMPERSONATE_SERVICE_ACCOUNT=--overwrite
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + shift 5
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + kubectl
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): kubectl controls the Kubernetes cluster manager.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Basic Commands (Beginner):
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): create Create a resource from a file or from stdin.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): run Run a particular image on the cluster
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): set Set specific features on objects
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Basic Commands (Intermediate):
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): explain Documentation of resources
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): get Display one or many resources
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): edit Edit a resource on the server
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): delete Delete resources by filenames, stdin, resources and names, or by resources and label selector
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Deploy Commands:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): rollout Manage the rollout of a resource
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): scale Set a new size for a Deployment, ReplicaSet or Replication Controller
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): autoscale Auto-scale a Deployment, ReplicaSet, StatefulSet, or ReplicationController
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Cluster Management Commands:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): certificate Modify certificate resources.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): cluster-info Display cluster info
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): top Display Resource (CPU/Memory) usage.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): cordon Mark node as unschedulable
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): uncordon Mark node as schedulable
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): drain Drain node in preparation for maintenance
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): taint Update the taints on one or more nodes
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Troubleshooting and Debugging Commands:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): describe Show details of a specific resource or group of resources
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): logs Print the logs for a container in a pod
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): attach Attach to a running container
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): exec Execute a command in a container
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): port-forward Forward one or more local ports to a pod
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): proxy Run a proxy to the Kubernetes API server
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): cp Copy files and directories to and from containers.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): auth Inspect authorization
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): debug Create debugging sessions for troubleshooting workloads and nodes
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Advanced Commands:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): diff Diff live version against would-be applied version
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): apply Apply a configuration to a resource by filename or stdin
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): patch Update field(s) of a resource
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): replace Replace a resource by filename or stdin
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): wait Experimental: Wait for a specific condition on one or many resources.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): kustomize Build a kustomization target from a directory or URL.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Settings Commands:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): label Update the labels on a resource
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): annotate Update the annotations on a resource
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): completion Output shell completion code for the specified shell (bash or zsh)
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Other Commands:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): api-resources Print the supported API resources on the server
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): api-versions Print the supported API versions on the server, in the form of "group/version"
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): config Modify kubeconfig files
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): plugin Provides utilities for interacting with plugins.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): version Print the client and server version information
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Usage:
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): kubectl [flags] [options]
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Use "kubectl <command> --help" for more information about a given command.
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): Use "kubectl options" for a list of global command-line options (applies to all commands).
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): + annotate --overwrite sa -n kube-system external-dns iam.gke.io/gcp-service-account=external-dns@development-319813.iam.gserviceaccount.com
module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): .terraform/modules/annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh: line 36: annotate: command not found
╷
│ Error: local-exec provisioner error
│
│ with module.annotate-sa.module.gcloud_kubectl.null_resource.run_command[0],
│ on .terraform/modules/annotate-sa/main.tf line 231, in resource "null_resource" "run_command":
│ 231: provisioner "local-exec" {
│
│ Error running command 'PATH=/google-cloud-sdk/bin:$PATH
│ .terraform/modules/annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh
│ dev-cluster us-central1 false false kubectl annotate --overwrite sa -n
│ kube-system external-dns
│ iam.gke.io/gcp-service-account=external-dns@"here goes workload identity".iam.gserviceaccount.com
│ ': exit status 127. Output: + '[' 12 -lt 5 ']'
│ + CLUSTER_NAME=dev-cluster
│ + LOCATION=`here goes the location`
│ + PROJECT_ID=false
│ + INTERNAL=false
│ + USE_EXISTING_CONTEXT=kubectl
│ + ENABLE_IMPERSONATE_SERVICE_ACCOUNT=annotate
│ + IMPERSONATE_SERVICE_ACCOUNT=--overwrite
│ + shift 5
│ + kubectl
│ kubectl controls the Kubernetes cluster manager.
│
│ Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/
│
│ Basic Commands (Beginner):
│ create Create a resource from a file or from stdin.
│ expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
│ run Run a particular image on the cluster
│ set Set specific features on objects
│
│ Basic Commands (Intermediate):
│ explain Documentation of resources
│ get Display one or many resources
│ edit Edit a resource on the server
│ delete Delete resources by filenames, stdin, resources and names, or by resources and label selector
│
│ Deploy Commands:
│ rollout Manage the rollout of a resource
│ scale Set a new size for a Deployment, ReplicaSet or Replication Controller
│ autoscale Auto-scale a Deployment, ReplicaSet, StatefulSet, or ReplicationController
│
│ Cluster Management Commands:
│ certificate Modify certificate resources.
│ cluster-info Display cluster info
│ top Display Resource (CPU/Memory) usage.
│ cordon Mark node as unschedulable
│ uncordon Mark node as schedulable
│ drain Drain node in preparation for maintenance
│ taint Update the taints on one or more nodes
│
│ Troubleshooting and Debugging Commands:
│ describe Show details of a specific resource or group of resources
│ logs Print the logs for a container in a pod
│ attach Attach to a running container
│ exec Execute a command in a container
│ port-forward Forward one or more local ports to a pod
│ proxy Run a proxy to the Kubernetes API server
│ cp Copy files and directories to and from containers.
│ auth Inspect authorization
│ debug Create debugging sessions for troubleshooting workloads and nodes
│
│ Advanced Commands:
│ diff Diff live version against would-be applied version
│ apply Apply a configuration to a resource by filename or stdin
│ patch Update field(s) of a resource
│ replace Replace a resource by filename or stdin
│ wait Experimental: Wait for a specific condition on one or many resources.
│ kustomize Build a kustomization target from a directory or URL.
│
│ Settings Commands:
│ label Update the labels on a resource
│ annotate Update the annotations on a resource
│ completion Output shell completion code for the specified shell (bash or zsh)
│
│ Other Commands:
│ api-resources Print the supported API resources on the server
│ api-versions Print the supported API versions on the server, in the form of "group/version"
│ config Modify kubeconfig files
│ plugin Provides utilities for interacting with plugins.
│ version Print the client and server version information
│
│ Usage:
│ kubectl [flags] [options]
│
│ Use "kubectl <command> --help" for more information about a given command.
│ Use "kubectl options" for a list of global command-line options (applies to
│ all commands).
│ + annotate --overwrite sa -n kube-system external-dns
│ iam.gke.io/gcp-service-account=external-dns@"here goes workload identity".iam.gserviceaccount.com
│ .terraform/modules/annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh:
│ line 36: annotate: command not found
│
╵
Releasing state lock. This may take a few moments...
ERRO[0049] 1 error occurred:
* exit status 1
gvasquez-plenty commented
@bharathkkb any update about this?
kherrmann3 commented
The module as defined within terraform looks like this:
module "annotate-sa" {
source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
version = "~> 2.1.0"
enabled = var.use_existing_k8s_sa && var.annotate_k8s_sa
skip_download = true
cluster_name = var.cluster_name
cluster_location = var.location
impersonate_service_account = var.impersonate_service_account
kubectl_create_command = "kubectl annotate --overwrite sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account=${local.gcp_sa_email}"
kubectl_destroy_command = "kubectl annotate sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account-"
}This is part of the log that I find interesting:
Creating...
Provisioning with 'local-exec'...
(local-exec): Executing: ["/bin/sh" "-c" "PATH=/google-cloud-sdk/bin:$PATH\n.terraform/modules/annotate-sa/modules/kubectl-wrapper/scripts/kubectl_wrapper.sh dev-cluster us-central1 false false kubectl annotate --overwrite sa -n kube-system e
(local-exec): + '[' 12 -lt 5 ']'
(local-exec): + CLUSTER_NAME=dev-cluster
(local-exec): + LOCATION=us-central1
(local-exec): + PROJECT_ID=false
(local-exec): + INTERNAL=false
(local-exec): + USE_EXISTING_CONTEXT=kubectl
(local-exec): + ENABLE_IMPERSONATE_SERVICE_ACCOUNT=annotate
(local-exec): + IMPERSONATE_SERVICE_ACCOUNT=--overwrite
(local-exec): + shift 5
(local-exec): + kubectl
(local-exec): kubectl controls the Kubernetes cluster manager.
For some reason the module appears to be constructing the "create_cmd_body" in a way that creates very odd values for ENABLE_IMPERSONATE_SERVICE_ACCOUNT and IMPERSONATE_SERVICE_ACCOUNT.
In our terraform module configuration we have this: impersonate_service_account = var.impersonate_service_account
And our default value there is "", so basically, this translates to impersonate_service_account = "". Where is the value for ENABLE_IMPERSONATE_SERVICE_ACCOUNT derived? I don't see it in the modules inputs.
There is a curious space, after us-central1, in this "create_cmd_body": dev-cluster us-central1 false false kubectl annotate --overwrite sa -n kube-system.... It makes me think there is a "" value being injected in here someplace, throwing off the arguments placement.
gvasquez-flugel commented
If you don't specify impersonate_service_account is it going to take the default value and that is "" like you said , so I don't think that is the reason.
@kherrmann3 Is it an issue with our code or the module?
kherrmann3 commented
I'm trying to see how to get the "create_cmd_body" arguments to align with the expected parameters in the kubectl_wrapper.sh. So, I decided to hardcode the project_id value within the modules configuration within the main.tf.
module "annotate-sa" {
source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
version = "~> 2.1.0"
enabled = var.use_existing_k8s_sa && var.annotate_k8s_sa
skip_download = true
cluster_name = var.cluster_name
cluster_location = var.location
project_id = "project-x"
kubectl_create_command = "kubectl annotate --overwrite sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account=${local.gcp_sa_email}"
kubectl_destroy_command = "kubectl annotate sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account-"
}
And this is the output I'm seeing when it fails:
(local-exec): + '[' 11 -lt 5 ']'
(local-exec): + CLUSTER_NAME=dev-cluster
(local-exec): + LOCATION=us-central1
(local-exec): + PROJECT_ID=false
(local-exec): + INTERNAL=false
(local-exec): + USE_EXISTING_CONTEXT=kubectl
(local-exec): + ENABLE_IMPERSONATE_SERVICE_ACCOUNT=annotate
(local-exec): + IMPERSONATE_SERVICE_ACCOUNT=sa
(local-exec): + shift 5
(local-exec): + kubectl
(local-exec): kubectl controls the Kubernetes cluster manager.
As you can see, PROJECT_ID does not get the value "project-x". So, it seems to be having issues when it attempts to craft the "create_cmd_body". If it is in our configuration, I'm not sure what we are doing wrong.
kherrmann3 commented
Okay, it appears it might have been an issue with version 2.1.0, as I recently tested against 3.0.1 and it applied successfully.
That said, I feel like something might still be off, because I had tested 3.0.1 in the past (before I discovered we were missing the PROJECT_ID configuration in the modules block) and it didn't complain that PROJECT_ID was missing even though USE_EXISTING_CONTEXT wasn't set, which I though PROJECT_ID is conditional on USE_EXISTING_CONTEXT.
To sum up...I believe the combination of these two items fixed the issue for us:
added PROJECT_ID = var.project_id
updated version to ~> 3.0.1
github-actions commented
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days