GCP Secure CI/CD Pipeline Demo

This repo bootstraps a CI/CD pipeline on Google Cloud to demonstrate policy controls for container image provenance, and for GKE authorization based on attestation. What's included:

Cloud Build pipeline with on tag GitHub repo trigger and SBOM generation
Test and Prod GKE clusters configured for Binary Authorization
2 Binary Authorization policies to ensure that images:
- Have Cloud Build provenance (been built in GCB)
- Meet minimal vulnerability validation policy (been successfully scanned using Kritis)
1 vulnerability signing policy with min fixable and un-fixable settings
Artifact Registry registry with SLSA level 3 verification
Container analysis with vulnerability scanning and meta-data management
Cloud Deploy pipeline with approval-based test to prod promotion
GKE security posture dashboard with Configuration concerns

Requirements

gcloud
jq

Setup

Setup Environment

Start by forking this repo into your GitHub Account. Navigate to https://github.com/mchmarny/sds-demo and click fork.

Next, clone that repo locally:

git clone git@github.com:<your-github-username>/sds-demo.git
cd sds-demo

Provision Resources

To deploy this pipeline into your GCP project, you will also need to export the ID of target project:

export PROJECT_ID=<your-project-id-here>

Next, create the required GCP resources (KMS, Artifact Registry repo, and service account policies):

setup/init

Create GKE cluster:

setup/clusters

And, configure Binary Authorization:

setup/binauthz

Next, trust all Google-provided system images in UI (bottom):

https://console.cloud.google.com/security/binary-authorization/policy/edit

Expand Additional settings for GKE and Anthos deployments
Check Trust all Google-provided system images
Click SAVE POLICY

Still have not been able to figure out how to enable this programmatically.

Wait for cluster to be created:

When ready, status will change from PROVISIONING to RUNNING

watch gcloud container clusters list --filter="resourceLabels.demo:build"

Configure GCB Trigger

The OAuth bit can't be scripted, so we have to do this manually.

Navigate to the triggers page

https://console.cloud.google.com/cloud-build/triggers

Make sure to set the region to your region (default: us-west1) at the top left of the screen.

Go through the Connect Repository flow. The important bits:

Select source: GitHub (Cloud Build GitHub App)
Select repository
- GitHub Account: this is your GitHub Username (should be populated after successful auth)
- Repository: the newly cloned repo (your-github-username/sds-demo)
Create a trigger: click DONE (we will script that part next)

Create Trigger

Create the GCP worker pool and the trigger on tag in the cloned app repo:

Make sure to replace the your-github-username with your GitHub username.

setup/trigger <your-github-username>

Check that the trigger has been created:

https://console.cloud.google.com/cloud-build/triggers

Enable Permissions

Finally, navigate to GCB settings, and enable following permissions:

This is used in the demo later.

Kubernetes Engine
Cloud KMS
Service Accounts
Cloud Build

https://console.cloud.google.com/cloud-build/settings/service-account

Demo

Attestation Validation

Deploy sample image that was not built in GCP. First, get the configuration with which the demo clusters were configured:

. setup/config

Next, get the test cluster credentials:

gcloud container clusters get-credentials demo-test --region $CLUSTER_ZONE

Then apply command will work because Kubernetes is declarative but as we will see in a minute, the workflow will fail to deploy.

kubectl apply -f test/non-gcp-built-image.yaml

Navigate to GKE workloads, and show how non-gcp-built-image failed to deploy due to lack of attestation (use cluster/namespace filters if needed).
- The error should have names of the two attestors that valid deployments need to have in this cluster: built-by-cloud-build and vulnz-attestor
Navigate to Security Policy and show BinAuth Policy rules
- Vulnerability attestation (scanned and signed with KMS key)
- Built in Cloud Build (attested with KMS key signature)
- Show dry run and images exempt option on policy as a means to incremental rollout strategy

Build on Tag (end-to-end demo)

Show delivery pipeline config app/clouddeploy.yaml
Make some code change
- Make sure you bump version number in app/.version
Git add, commit, push:

git add --all
git commit -m 'demo'
git push --all

Git tag and push, to trigger the GCB pipeline:

export VERSION_TAG=$(cat app/.version)
git tag -s -m "demo" $VERSION_TAG
git push origin $VERSION_TAG

Navigate to Cloud Build triggers
- Push on tag (pattern)
- Back in code, review config app/cloudbuild.yaml
- Review the vulnerability scanner policy policy/vulnz-signing-policy.yaml

Navigate to Cloud Build builds in UI
- Drill on active build
- Review steps (show sbom, scan, and attest running concurrently):
  - test - tests the code one more time
  - build - builds a docker image
  - publish - publishes that image to Artifact Registry
  - sbom - generates Software Bill of Materials for the published image
  - scan - scans image for culnerabilities (note: this step will fail if scanner finds vulnerabilities with severity higher than the max defined in policy/vulnz-signing-policy.yaml)
  - attest - creates and publishes attestation that this iamge was built in GCB using KMS key
  - release - deploys the image via Cloud Deploy
- On Build Summary, show Build Artifacts > Image security insights (SLSA Build Level 3)
  - Show Dependencies and Build Provenance YAMLs
Navigate to Artifact Registry list of registries
- Drill into sds-demo/hello
- Show attestation and signature artifacts (*.att and *.sig)
- Navigate to Manifest in *.sig, show cosign/signature
- Navigate to the image (the one with the v* tag) and show Vulnerabilities
- Show policy
- Show Kritis Signer step in cloudbuild.yaml
- Show in-toto attestation in CLI:

Retrieve provenance as JSON:

gcloud artifacts docker images describe $digest --show-provenance --format json > provenance.json

Verify provenance:

Using slsa-verifier

slsa-verifier verify-image $digest \
  --provenance-path provenance.json \
  --source-uri https://github.com/mchmarny/sds-demo \
  --builder-id https://cloudbuild.googleapis.com/GoogleHostedWorker@v0.3

Note, source-tag tag is not supported in GCB verification. The source-uri and builder-id come from materials.uri and builder.id respectively in the in-toto statement (provenance_summary.provenance.build.intotoStatement).

Extract the SBOM attestation (update the key location as needed)

cosign verify-attestation --type spdxjson \
  --key gcpkms://projects/cloudy-demos/locations/us-west1/keyRings/binauthz/cryptoKeys/binauthz-signer/cryptoKeyVersions/1 \
  $digest | jq -r '.payload' | base64 -d | jq .

Make sure the Cloud Build job completed before proceeding.

Navigate to Cloud Deploy pipelines
- Drill into deploy-demo-pipeline
- Show 2 environments (test, prod)
- Drill into release, rollouts, and targets

Navigate to GKE workloads
- Drill into hello (note non-gcp-built-image erred due to lack of attestation)
- Navigate to exposed hello service at the bottom
- Click on endpoint, and show version using /api/ping (should be same as the version in .version file in the repo)

Back to Cloud Deploy pipelines
- Show promotion, review, and approval with manifest diffs and annotation comp (show more)

Navigate to Binary Authorization policy
- Show cluster specific rules, and edit policy
- Show Custom exemption rules
- Show GKE cluster-specific rules

Show GKE Security Posture
- Show concerns (note: there may not be much, if you are using brand new project for this demo)

Cleanup

setup/delete

Disclaimer

This is my personal project and it does not represent my employer. While I do my best to ensure that everything works, I take no responsibility for issues caused by this code.