How to specify Workload Identity for GKE?

[samkhawase]

I'm trying to use this Helm chart to deploy on GKE, but I can't seem to connect the Metabase with CloudSQL DB.
Hi,

I also have the setup the Workload Identity and connected the Google Service Account to the Kubernetes Service Accounts. However the Metabase Helm install fails and couldn't find the CloudSQL DB.

Where do I specify the Workload Identity?

Here's the section for the Backend DB configuration from my values.yml

# Backend database
database:
  # Database type (h2 / mysql / postgres), default: h2
  type: postgres
  ## Only need when you use mysql / postgres
  host: <IP Address masked>
  port: 5432
  dbname: metabase
  username: metabase
  password: <Password masked>
  ## One or more Google Cloud SQL database instances can be made available to Metabase via the *Cloud SQL Auth proxy*.
  ## These can be used for Metabase's internal database (by specifying `host: localhost` and the port above), or as
  ## additional databases (configured at Admin → Databases). Workload Identity should be used for authentication, so
  ## that when `serviceAccount.create=true`, `serviceAccount.annotations` should contain:
  ##   iam.gke.io/gcp-service-account: your-gsa@email
  ## Ref: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine
  googleCloudSQL:
    ## Found in Cloud Console "Cloud SQL Instance details" or using `gcloud sql instances describe INSTANCE_ID`
    ## example format: $project:$region:$instance=tcp:$port
    ## Each connection must have a unique TCP port.
    instanceConnectionNames: [<my-project:my-region:my-instance=tcp:port Masked>]
    ## Option to use a specific version of the *Cloud SQL Auth proxy* sidecar image.
    ## ref: https://console.cloud.google.com/gcr/images/cloudsql-docker/GLOBAL/gce-proxy
    sidecarImageTag: latest
    ## ref: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#running_the_as_a_sidecar
    resources:
      {}

[pmint93]

Hi @samkhawase

Does it fail on command helm install ... or metabase failed at runtime which unable to connect to Google Cloud SQL ?

To enable Workload Identity: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#workload-identity

To specify workload identity, use serviceAccount with annotations here: https://github.com/pmint93/helm-charts/blob/master/charts/metabase/values.yaml#L170

[samkhawase]

Thanks @pmint93

I tried adding the Google Service account name and it gave me the following error:

Error: INSTALLATION FAILED: Deployment.apps "metabase-helm" is invalid: spec.template.spec.serviceAccountName:
Invalid value: "my-team@my-project-stage.iam.gserviceaccount.com": a DNS-1123 subdomain must consist of lower case
alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character 
(e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'

I tried adding the K8s Service Account name which reads like my-project-ksa, and it fails to get the KSA from the default namespace. How do we specify the namespace. I tried my-namespace/my-project-ksa but it didn't work.

[pmint93]

The first error during helm install indicate that service name must follow DNS-1123 validation (it will be used as hostname so cannot contain characters like "@")

According to this guide Workload Identity, you basically do following steps by order

1. Enable Workload Identity on your GKE cluster

2. Deploy metabase with whatever service name you like (at this point metabase still unable to connect to Google Cloud SQL until step 4)

3. add-iam-policy-binding on Google Cloud which bind the KSA & GSA together

4. Annotate the service just created after step (1)

You may annotate the service right at the step (1), but I'm not sure if it work

[samkhawase]

@pmint93

I did setup a GSA -> KSA mapping through workload identity and it still doesn't work. I get the following error when I install the helm chart with the custom values.yml file.

Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/my-project-dev/instances/europe-west3~metabase?alt=json&prettyPrint=false": 
metadata: GCE metadata "instance/service-accounts/default/token?scopes=https//www.googleapis.com/auth/sqlservice.admin" not defined

[samkhawase]

I managed to resolve the issue by commenting the googleCloudSQL sidecar section. I wonder if it's needed if we're using workload identity

[pmint93]

It's related to Google Cloud products, you may want to contact Google Cloud supports or Google Cloud experts to debug your issue.

The proxy sidecar is not a must but a recommended way https://cloud.google.com/sql/docs/postgres/sql-proxy