DevSecOpsSamples/gke-workload-identity

GKE IAM role and workload identity sample project

GKE Workload Identity

Overview

The Workload Identity is the recommended way for your workloads running on Google Kubernetes Engine (GKE) to access Google Cloud services in a secure and manageable way. In this sample project, we will learn GKE security with the IAM service account and Workload Identity.

Applications running on GKE might need access to Google Cloud APIs such as Compute Engine API, BigQuery Storage API, or Machine Learning APIs. Workload Identity allows a Kubernetes service account in your GKE cluster to act as an IAM service account. Pods that use the configured Kubernetes service account automatically authenticate as the IAM service account when accessing Google Cloud APIs. Using Workload Identity allows you to assign distinct, fine-grained identities and authorization for each application in your cluster.

https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity

Objectives

Learn the features below:

• Three steps for Workload Identity
  • Creating an IAM service account and Kubernetes service account
  • IAM policy binding between IAM service account and Kubernetes service account
  • Annotate the Kubernetes service account
• Pod specification for GKE service account and GCP load balancer
• Create resources with Terraform

Table of Contents

• Prerequisites
• Step1: Create a GKE cluster
• Step2: Create Kubernetes namespaces and service accounts
• Step3: IAM service account for bucket-api
  • 3.1. Creating an IAM service account
  • 3.2. IAM policy binding between IAM service account and Kubernetes service account
  • 3.3. Annotate the Kubernetes service account
• Step4: Deploy bucket-api
• Step5: IAM service account for pubsub-api
  • 5.1. Creating an IAM service account
  • 5.2. IAM policy binding between IAM service account and Kubernetes service account
  • 5.3. Annotate the Kubernetes service account
• Step6: Create a Topic/Subscription and grant a permission
  • 6.1. Create a Topic and Subscription
  • 6.2. Grant permission to IAM service account to publish to Topic
  • 6.3. Grant permission to IAM service account for subscription
• Step7: Deploy pubsub-api
• Unittest
• Structure
• Terraform
• Troubleshooting
• Cleanup

---

Prerequisites

Installation

• Install the gcloud CLI
• Install gsutil
• Install kubectl and configure cluster access

Set environment variables

# echo "export PROJECT_ID=<your-project-id>" >> ~/.bashrc
PROJECT_ID="<your-project-id>"
COMPUTE_ZONE="us-central1"
SERVICE_ACCOUNT="bucket-api-sa"
PUBSUB_SERVICE_ACCOUNT="pubsub-api-sa"
GCS_BUCKET_NAME="${PROJECT_ID}-bucket-api"

	Environment Variable	Value	Description
1	PROJECT_ID	sample-project	This variable will also be used for pub/sub deployment.
2	COMPUTE_ZONE	us-central1	Run gcloud compute zones list to get all zones.
3	SERVICE_ACCOUNT	bucket-api-sa	IAM service account for bucket-api to access to GCS bucket only.
4	PUBSUB_SERVICE_ACCOUNT	pubsub-api-sa	IAM service account for pubsub-api to access to pub/sub only.
5	GCS_BUCKET_NAME	bucket-api

Set GCP project

gcloud config set project ${PROJECT_ID}
gcloud config set compute/zone ${COMPUTE_ZONE}

---

Step1: Create a GKE cluster

Create an Autopilot GKE cluster. Autopilot clusters must be regional clusters and it may take around 9 minutes.

CLUSTER_REGION="us-central1" 
gcloud container clusters create-auto sample-cluster-dev \
       --region=${CLUSTER_REGION} --project ${PROJECT_ID}

NAME                LOCATION     MASTER_VERSION  MASTER_IP        MACHINE_TYPE  NODE_VERSION    NUM_NODES  STATUS
sample-cluster-dev  us-central1  1.24.5-gke.600  xxx.xxx.xxx.xxx  e2-medium     1.24.5-gke.600  3          RUNNING

gcloud container clusters get-credentials sample-cluster-dev \
       --region=${COMPUTE_ZONE} --project ${PROJECT_ID}

If you want to use a Standard mode cluster instead of Autopilot GKE cluster. Refer to the README-standard-cluster.md.

Step2: Create Kubernetes namespaces and service accounts

API	Object	Name	Description
bucket-api	namespace	bucket-api-ns
bucket-api	service account	bucket-api-ksa	Kubernetes service account
pubsub-api	namespace	pubsub-api-ns
pubsub-api	service account	pubsub-api-ksa	Kubernetes service account

kubectl create namespace bucket-api-ns
kubectl create namespace pubsub-api-ns

kubectl create serviceaccount --namespace bucket-api-ns bucket-api-ksa
kubectl create serviceaccount --namespace pubsub-api-ns pubsub-api-ksa

Step3: IAM service account for bucket-api

3.1. Creating an IAM service account.

echo "PROJECT_ID: ${PROJECT_ID}, SERVICE_ACCOUNT: ${SERVICE_ACCOUNT}"

gcloud iam service-accounts create ${SERVICE_ACCOUNT} --display-name="bucket-api-ns service account"
gcloud iam service-accounts list | grep bucket-api-sa

3.2. Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts.

gcloud iam service-accounts add-iam-policy-binding \
       --role roles/iam.workloadIdentityUser \
       --member "serviceAccount:${PROJECT_ID}.svc.id.goog[bucket-api-ns/bucket-api-ksa]" \
       ${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com

Updated IAM policy for serviceAccount [bucket-api@sample-project.iam.gserviceaccount.com].
bindings:
- members:
  - serviceAccount:sample-project.svc.id.goog[bucket-api-ns/bucket-api-ksa]
  role: roles/iam.workloadIdentityUser
etag: BwXtbNaPnNg=
version: 1

3.3. Annotate the Kubernetes service account with the email address of the IAM service account.

kubectl annotate serviceaccount --namespace bucket-api-ns bucket-api-ksa \
        iam.gke.io/gcp-service-account=${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com

3.4. GCS bucket creation and grant a permission.

Create a GCS bucket

gcloud storage buckets create gs://${GCS_BUCKET_NAME}

Grant objectAdmin role to IAM service account to access a GCS bucket.

gsutil iam ch serviceAccount:${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com:objectAdmin \
       gs://${GCS_BUCKET_NAME}/

Refer to the https://cloud.google.com/storage/docs/access-control/iam-roles page for predefined roles.

Use serviceAccountName for Pods:

src/bucket-api/bucket-api-template.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bucket-api
  namespace: bucket-api-ns
  annotations:
    app: 'bucket-api'
spec:
  replicas: 2
  selector:
    matchLabels:
      app: bucket-api
  template:
    metadata:
      labels:
        app: bucket-api
    spec:
      serviceAccountName: bucket-api-ksa
      containers:
        - name: bucket-api

NOTE: If you want to test for debugging on your desktop with the IAM key, refer to the README-test.md.

Step4: Deploy bucket-api

4.1. Build and push to GCR:

cd src/bucket-api

docker build -t bucket-api . --platform linux/amd64
docker tag bucket-api:latest gcr.io/${PROJECT_ID}/bucket-api:latest

gcloud auth configure-docker
docker push gcr.io/${PROJECT_ID}/bucket-api:latest

4.2. Create and deploy K8s Deployment, Service, HorizontalPodAutoscaler, Ingress, and GKE BackendConfig using a template file.

sed -e "s|<project-id>|${PROJECT_ID}|g" bucket-api-template.yaml > bucket-api.yaml
cat bucket-api.yaml
kubectl apply -f bucket-api.yaml

Confirm that pod configuration and logs after deployment:

kubectl describe pods -n bucket-api-ns
kubectl logs -l app=bucket-api -n bucket-api-ns

4.3 Invoke /bucket API using a load balancer IP:

LB_IP_ADDRESS=$(gcloud compute forwarding-rules list | grep bucket-api | awk '{ print $2 }' | head -n 1)
echo "http://${LB_IP_ADDRESS}/" 
curl http://${LB_IP_ADDRESS}/
curl http://${LB_IP_ADDRESS}/bucket

{"host":"34.149.214.247","message":"bucket-api","method":"GET","url":"http://34.149.214.247/"}

{"blob_name":"put-test.txt","bucket_name":"bucket-api","response":"read/write test, bucket: bucket-api"}

Step5: IAM service account for pubsub-api

5.1. Creating an IAM service account.

echo "PROJECT_ID: ${PROJECT_ID}, PUBSUB_SERVICE_ACCOUNT: ${PUBSUB_SERVICE_ACCOUNT}"

gcloud iam service-accounts create ${PUBSUB_SERVICE_ACCOUNT} --display-name="pubsub-api-ns service account"
gcloud iam service-accounts list | grep pubsub-api

5.2. Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts.

gcloud iam service-accounts add-iam-policy-binding \
       --role roles/iam.workloadIdentityUser \
       --member "serviceAccount:${PROJECT_ID}.svc.id.goog[pubsub-api-ns/pubsub-api-ksa]" \
       ${PUBSUB_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com

Updated IAM policy for serviceAccount [pubsub-api@sample-project.iam.gserviceaccount.com].
bindings:
- members:
  - serviceAccount:sample-project.svc.id.goog[pubsub-api-ns/pubsub-api-ksa]
  role: roles/iam.workloadIdentityUser
etag: BwXtbNaPnNg=
version: 1

5.3. Annotate the Kubernetes service account with the email address of the IAM service account.

kubectl annotate serviceaccount --namespace pubsub-api-ns pubsub-api-ksa \
        iam.gke.io/gcp-service-account=${PUBSUB_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com

Step6: Create a Topic/Subscription and grant a permission

6.1. Create a Topic and Subscription.

gcloud services enable cloudresourcemanager.googleapis.com pubsub.googleapis.com \
       container.googleapis.com
gcloud pubsub topics create echo
gcloud pubsub subscriptions create echo-read --topic=echo

6.2. Grant permission to IAM service account to publish to Topic.

echo "PUBSUB_SERVICE_ACCOUNT: ${PUBSUB_SERVICE_ACCOUNT}"
gcloud pubsub topics add-iam-policy-binding echo  \
      --member=serviceAccount:${PUBSUB_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com \
      --role=roles/pubsub.publisher

gcloud pubsub topics get-iam-policy echo --format yaml

bindings:
- members:
  - serviceAccount:pubsub-api@sample-project.iam.gserviceaccount.com
  role: roles/pubsub.publisher
etag: BwXtaNHVrCw=
version: 1

6.3. Grant permission to IAM service account for subscription.

gcloud pubsub subscriptions add-iam-policy-binding echo-read \
       --member=serviceAccount:${PUBSUB_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com  \
       --role=roles/pubsub.subscriber

gcloud pubsub subscriptions get-iam-policy projects/${PROJECT_ID}/subscriptions/echo-read \
       --format yaml

bindings:
- members:
  - serviceAccount:pubsub-api@sample-project.iam.gserviceaccount.com
  role: roles/pubsub.subscriber
etag: BwXtaNHVrCw=
version: 1

Step7: Deploy pubsub-api

Build and push to GCR:

cd ../pubsub-api
docker build -t pubsub-api . --platform linux/amd64
docker tag pubsub-api:latest gcr.io/${PROJECT_ID}/pubsub-api:latest

gcloud auth configure-docker
docker push gcr.io/${PROJECT_ID}/pubsub-api:latest

Create and deploy K8s Deployment, Service, HorizontalPodAutoscaler, Ingress, and GKE BackendConfig using a template file.

sed -e "s|<project-id>|${PROJECT_ID}|g" pubsub-api-template.yaml > pubsub-api.yaml
cat pubsub-api.yaml
kubectl apply -f pubsub-api.yaml -n pubsub-api-ns

Confirm that pod configuration and logs after deployment:

kubectl describe pods -n pubsub-api-ns
kubectl logs -l app=pubsub-api -n pubsub-api-ns
#kubectl get service -n pubsub-api-ns -o yaml
kubectl describe service -n pubsub-api-ns

Confirm that response of /pub, /sub, and /bucket APIs.

LB_IP_ADDRESS=$(gcloud compute forwarding-rules list | grep pubsub-api | awk '{ print $2 }' | head -n 1)
echo ${LB_IP_ADDRESS}

curl http://${LB_IP_ADDRESS}/pub

{
  "result": "6237829865389825",
  "topic_name": "projects/sample-project/topics/echo"
}

curl http://${LB_IP_ADDRESS}/sub

{
  "acknowledged": 7,
  "subscription_path": "projects/sample-project/subscriptions/echo-read",
  "topic_name": "projects/sample-project/topics/echo"
}

curl http://${LB_IP_ADDRESS}/bucket

/bucket API does not work because permission granted to pub/pub service only.

Unittest

src/README.md

Structure

├── build.gradle
├── pytest.ini
├── requirements.txt
├── src
│   ├── all-commands.sh
│   ├── bucket-api
│   │   ├── Dockerfile
│   │   ├── bucket-api-template.yaml
│   │   ├── deploy.sh
│   │   ├── main.py
│   │   ├── requirements.txt
│   │   └── tests
│   │       └── test_bucket_api.py
│   ├── pubsub-api
│   │   ├── Dockerfile
│   │   ├── deploy.sh
│   │   ├── pubsub-api-template.yaml
│   │   ├── pubsub_api_main.py
│   │   ├── requirements.txt
│   │   └── test_pubsub_api.py
│   └── tf

• bucket-api-template.yaml
• pubsub-api-template.yaml

Terraform

If you use the Terraform, you can create all resources Terraform at a time. Please refer to the src/tf/README.md page.

Troubleshooting

• GCS bucket permission error

  Create a key using gcloud iam service-accounts keys create command and set the GOOGLE_APPLICATION_CREDENTIALS environment variable on you deksop. For details refer to the README-test.md.

  If working fine with the credential file, check node option with README-standard-cluster.md file.

Cleanup

kubectl delete -f bucket-api/bucket-api.yaml
kubectl delete -f pubsub-api/pubsub-api.yaml

kubectl delete namespace bucket-api
kubectl delete namespace pubsub-api

gcloud storage buckets delete gs://${GCS_BUCKET_NAME}

gcloud pubsub subscriptions remove-iam-policy-binding echo-read --member=serviceAccount:${PUBSUB_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/pubsub.subscriber
gcloud pubsub subscriptions delete echo-read
gcloud pubsub topics delete echo

gcloud iam service-accounts delete "${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" 
gcloud iam service-accounts delete "${PUBSUB_SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" 

docker system prune -a

References

• Use Workload Identity
• Cloud SDK > Documentation > Reference > gcloud iam service-accounts add-iam-policy-binding
• Cloud Pub/Sub > Documentation > Samples > Subscribe with synchronous pull
• https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
• Enabling IAP for GKE