Setting `project_id` does not work to overwrite the project of the service account which is getting the credentials

[moritzschmitz-oviva]

TL;DR

Workload identity provider lives in a different project than the GKE cluster I want to access from a GitHub workflow. get-gke-credentials tries to access the GKE cluster in the project of the workload identity provider. Service account also lives in the workload identity provider project.

Expected behavior

project_id should be correctly picked up and overwrite any other projects (e.g. service account project)

Observed behavior

project_id is not respected. It seems that always the project ID of the service account is used

Action YAML

name: Deploy service

on:
  workflow_dispatch:
  push:
    branches:
      - main
    paths:
      - 'it/service/**'
      - 'it/k8s/service/**'

env:
  PROJECT_ID: "project-one"
  GKE_CLUSTER: "project-one-gke"
  LOCATION: "europe-west3-b"
  PROVIDER_NAME: "projects/NUMBER-OF-PROJECT-TWO/locations/global/workloadIdentityPools/github-actions-sa/providers/github-actions-sa"
  SA_EMAIL: "gh-wi-project-one@project-two.iam.gserviceaccount.com"

jobs:
  push-to-it:
    name: Push service
    runs-on: ubuntu-latest

    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      - name: 'Checkout Config repo'
        uses: actions/checkout@v2
        with:
          path: config-repo
      - name: 'Checkou Oviva Github Actions'
        uses: actions/checkout@v2
        with:
          repository: oviva-ag/actions
          path: actions
          ssh-key: ${{ secrets.GH_ACTIONS_REPO_KEY }}
      - name: 'Checkout Helm Charts'
        uses: actions/checkout@v2
        with:
          repository: oviva-ag/k8s-test
          path: k8s
          token: ${{ secrets.GH_CI_PAT }}
      - name: 'Google auth'
        uses: google-github-actions/auth@v0
        with:
          workload_identity_provider: ${{ env.PROVIDER_NAME }}
          service_account: ${{ env.SA_EMAIL }}
      - name: 'Setup kubectl'
        uses: google-github-actions/get-gke-credentials@v0
        with:
          project_id: ${{ env.PROJECT_ID }}
          cluster_name: ${{ env.GKE_CLUSTER }}
          location: ${{ env.LOCATION }}
      - name: 'Install Helm Chart'
        uses: './actions/it-k8s-deploy'
        with:
          helm-path: 'k8s/helm'
          namespace: 'it'
          chart-name: 'service'
          release-name: 'service'
          config-repo-path: 'config-repo'
          value-files: 'k8s/helm/it-values.yaml config-repo/it/k8s/service/values.yaml'
      - name: 'Notify developers'
        env: 
          CHAT_WEBHOOK: ${{ secrets.GOOGLE_CHAT_NOTIFICATION_CHANNEL }}
        run: |
          curl -H "Content-Type: application/json" -X POST -d '{ "text":"Deployed service" }' "$CHAT_WEBHOOK"

Log output

Error: google-github-actions/get-gke-credentials failed with: kubernetes Engine API has not been used in project NUMBER-OF-PROJECT-TWO before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/container.googleapis.com/overview?project=NUMBER-OF-PROJECT-TWO then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Additional information

Project one owns the cluster.
Project two owns the service account to access the cluster and also the workload identity provider.

The service account in project two has roles/container.developer access in project one.

No matter which environment variable I overwrite, the logs always show that get-gek-credentials is trying to fetch credentials from project two. I tried all of the environment variables mentioned here: https://github.com/google-github-actions/auth#usage => export_environment_variables.

[moritzschmitz-oviva]

I don't really understand it, but I just had to enable the Kubernetes API in project two (workload identity provider project) for this to work.