GoogleCloudPlatform/alloydb-auth-proxy

Question: alloydb-auth-proxy as a service + GKE workload identity

Closed this issue · 4 comments

abezard-conduit commented

Question

Hi! Sorry that might be a dumb question but I cannot get the alloydb-auth-proxy to work with GKE workload identity when running as a k8s service.

I have the following deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: alloydb-auth-proxy
  name: alloydb-auth-proxy
spec:
  selector:
    matchLabels:
      app: alloydb-auth-proxy
  template:
    metadata:
      labels:
        app: alloydb-auth-proxy
    spec:
      serviceAccountName: alloydb-access
      containers: 
      - name: alloydb-auth-proxy
        image: us.gcr.io/alloydb-connectors/alloydb-auth-proxy:1.7.1
        ports:
        - containerPort: 5000
          protocol: TCP
        - containerPort: 5001
          protocol: TCP
        command: ["/alloydb-auth-proxy"]
        args:
        - "--auto-iam-authn"
        - "projects/{{ project }}/locations/us-west1/clusters/{{ cluster.name }}/instances/primary?port=5000"
        - "projects/{{ project }}/locations/us-west1/clusters/{{ cluster.name }}/instances/readpool?port=5001"
        - "--address=0.0.0.0"

The following k8s service:

---
apiVersion: v1
kind: Service
metadata:
  name: alloydb-auth-proxy
  namespace: alloydb-auth-proxy
spec:
  ports:
  - name: alloydb-auth-proxy-primary
    port: 5000
    protocol: TCP
    targetPort: 5000
  - name: alloydb-auth-proxy-readpool
    port: 5001
    protocol: TCP
    targetPort: 5001
  selector:
    app: alloydb-auth-proxy
  type: ClusterIP

The following k8s Service Account:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: alloydb-access@{{ project }}.iam.gserviceaccount.com
  name: alloydb-access
  namespace: alloydb-auth-proxy

The alloydb-access GCP Service Account has what I believe are the right IAM permissions:

  • Cloud AlloyDB Client
  • Service Usage Consumer

I have a user set in my DB with the alloydbiamuser IAM role and named alloydb-access@{{ project }}.iam as the documentation states:

For an IAM service account, supply the service account's address without the .gserviceaccount.com suffix. For example, to specify the service account my-service@my-project.iam.gserviceaccount.com, you would use the value my-service@my-project.iam here.

When I try to authenticate with alloydb through the alloydb-auth-proxy service using the postgres default user, everything works like a charm, when I try to use IAM authentication I get the following error:

# psql -h alloydb-auth-proxy -p 5000 -U alloydb-access@{{ project }} .iam
psql: error: connection to server at "alloydb-auth-proxy" (10.76.12.23), port 5000 failed: FATAL:  AlloyDB IAM user authentication failed for user "alloydb-access@{{ project }}.iam"

the alloydb.iam_authentication flag is set on my alloydb instances and I have rechecked all the bindings multiple times.

Any help will be greatly appreciated as I'm kind of lost right now.

Code

No response

Additional Details

  • I have enabled debug logs, but the alloydb-auth-proxy logs do not provide much information:
2024/03/07 03:41:58 [{{ project }}.us-west1.{{ cluster.name}}.primary] accepted connection from 10.72.6.63:38128
2024/03/07 03:42:22 {{ project }}.us-west1.{{ cluster.name}}.primary] client closed the connection
jackwotherspoon commented

Hi @abezard-conduit Thanks for the detailed information and great question! 😄

Because you mentioned everything works fine for the postgres user but not the IAM database user it makes me think there might be an IAM permissions missing.

The alloydb-access GCP Service Account has what I believe are the right IAM permissions:

  • Cloud AlloyDB Client
  • Service Usage Consumer

It looks like your service account may be missing the Cloud AlloyDB Database User IAM role? I would try granting it and seeing if that fixes your issue 😄

Happy to update any docs if this role fixes the issue and is missing from any pages.

abezard-conduit commented

hey @jackwotherspoon !
Thanks for getting back to me.

I guess what confuses me is:

  • in this documentation it's said that the auth-proxy needs to roles/alloydb.client permissions.
  • in this documentation it's said that the DB client needs roles/alloydb.databaseUser permissions.

From what I understand alloydb-auth-proxy is the component that takes care of authenticating and connecting to the Database, does it mean alloydb-auth-proxy needs both roles/alloydb.client / roles/alloydb.databaseUser permissions?

Or does it mean alloydb-auth-proxy needs roles/alloydb.client and the client talking to alloydb-auth-proxy needs roles/alloydb.databaseUser?

Thanks in advance!

jackwotherspoon commented

Hi @abezard-conduit let me try and clarify the above for you.

  • roles/alloydb.client : This role is required for all connections, when using built-in database authn (regular user and password) or IAM database authn. This role allows access to call the AlloyDB APIs to get the IP address of the instance and generate an ephemeral certificate to be used for the SSL connection.
  • roles/alloydb.databaseUser : Additional role ONLY required for IAM database authn. This role is required only for IAM database authn to allow the IAM user or service account to login to the database.

Hope this clarifies things, I'll look at the docs to see if we can improve the messaging and make things more clear in our documentation.

abezard-conduit commented

Hi @jackwotherspoon!

ok that makes so much more sense, thanks for your help.