/sds-demo

Software Delivery Shield demo illustrating end-to-end solution for secure software supply chain

Primary LanguageShellApache License 2.0Apache-2.0

GCP Secure CI/CD Pipeline Demo

This repo bootstraps a CI/CD pipeline on Google Cloud to demonstrate policy controls for container image provenance, and for GKE authorization based on attestation. What's included:

  • Cloud Build pipeline with on tag GitHub repo trigger and SBOM generation
  • Test and Prod GKE clusters configured for Binary Authorization
  • 2 Binary Authorization policies to ensure that images:
    • Have Cloud Build provenance (been built in GCB)
    • Meet minimal vulnerability validation policy (been successfully scanned using Kritis)
  • 1 vulnerability signing policy with min fixable and un-fixable settings
  • Artifact Registry registry with SLSA level 3 verification
  • Container analysis with vulnerability scanning and meta-data management
  • Cloud Deploy pipeline with approval-based test to prod promotion
  • GKE security posture dashboard with Configuration concerns

Requirements

Setup

Setup Environment

Start by forking this repo into your GitHub Account. Navigate to https://github.com/mchmarny/sds-demo and click fork.

Next, clone that repo locally:

git clone git@github.com:<your-github-username>/sds-demo.git
cd sds-demo

Provision Resources

To deploy this pipeline into your GCP project, you will also need to export the ID of target project:

export PROJECT_ID=<your-project-id-here>

Next, create the required GCP resources (KMS, Artifact Registry repo, and service account policies):

setup/init

Create GKE cluster:

setup/clusters

And, configure Binary Authorization:

setup/binauthz

Next, trust all Google-provided system images in UI (bottom):

https://console.cloud.google.com/security/binary-authorization/policy/edit

  • Expand Additional settings for GKE and Anthos deployments
  • Check Trust all Google-provided system images
  • Click SAVE POLICY

Still have not been able to figure out how to enable this programmatically.

Wait for cluster to be created:

When ready, status will change from PROVISIONING to RUNNING

watch gcloud container clusters list --filter="resourceLabels.demo:build"

Configure GCB Trigger

The OAuth bit can't be scripted, so we have to do this manually.

Navigate to the triggers page

https://console.cloud.google.com/cloud-build/triggers

Make sure to set the region to your region (default: us-west1) at the top left of the screen.

Go through the Connect Repository flow. The important bits:

  • Select source: GitHub (Cloud Build GitHub App)
  • Select repository
    • GitHub Account: this is your GitHub Username (should be populated after successful auth)
    • Repository: the newly cloned repo (your-github-username/sds-demo)
  • Create a trigger: click DONE (we will script that part next)

Create Trigger

Create the GCP worker pool and the trigger on tag in the cloned app repo:

Make sure to replace the your-github-username with your GitHub username.

setup/trigger <your-github-username>

Check that the trigger has been created:

https://console.cloud.google.com/cloud-build/triggers

Enable Permissions

Finally, navigate to GCB settings, and enable following permissions:

This is used in the demo later.

  • Kubernetes Engine
  • Cloud KMS
  • Service Accounts
  • Cloud Build

https://console.cloud.google.com/cloud-build/settings/service-account

Demo

Attestation Validation

Deploy sample image that was not built in GCP. First, get the configuration with which the demo clusters were configured:

. setup/config

Next, get the test cluster credentials:

gcloud container clusters get-credentials demo-test --region $CLUSTER_ZONE

Then apply command will work because Kubernetes is declarative but as we will see in a minute, the workflow will fail to deploy.

kubectl apply -f test/non-gcp-built-image.yaml
  • Navigate to GKE workloads, and show how non-gcp-built-image failed to deploy due to lack of attestation (use cluster/namespace filters if needed).
    • The error should have names of the two attestors that valid deployments need to have in this cluster: built-by-cloud-build and vulnz-attestor
  • Navigate to Security Policy and show BinAuth Policy rules
    • Vulnerability attestation (scanned and signed with KMS key)
    • Built in Cloud Build (attested with KMS key signature)
    • Show dry run and images exempt option on policy as a means to incremental rollout strategy

Build on Tag (end-to-end demo)

  • Show delivery pipeline config app/clouddeploy.yaml
  • Make some code change
    • Make sure you bump version number in app/.version
  • Git add, commit, push:
git add --all
git commit -m 'demo'
git push --all
  • Git tag and push, to trigger the GCB pipeline:
export VERSION_TAG=$(cat app/.version)
git tag -s -m "demo" $VERSION_TAG
git push origin $VERSION_TAG
  • Navigate to Cloud Build triggers
    • Push on tag (pattern)
    • Back in code, review config app/cloudbuild.yaml
    • Review the vulnerability scanner policy policy/vulnz-signing-policy.yaml

  • Navigate to Cloud Build builds in UI

    • Drill on active build
    • Review steps (show sbom, scan, and attest running concurrently):
      • test - tests the code one more time
      • build - builds a docker image
      • publish - publishes that image to Artifact Registry
      • sbom - generates Software Bill of Materials for the published image
      • scan - scans image for culnerabilities (note: this step will fail if scanner finds vulnerabilities with severity higher than the max defined in policy/vulnz-signing-policy.yaml)
      • attest - creates and publishes attestation that this iamge was built in GCB using KMS key
      • release - deploys the image via Cloud Deploy
    • On Build Summary, show Build Artifacts > Image security insights (SLSA Build Level 3)
      • Show Dependencies and Build Provenance YAMLs
  • Navigate to Artifact Registry list of registries

    • Drill into sds-demo/hello
    • Show attestation and signature artifacts (*.att and *.sig)
    • Navigate to Manifest in *.sig, show cosign/signature
    • Navigate to the image (the one with the v* tag) and show Vulnerabilities
    • Show policy
    • Show Kritis Signer step in cloudbuild.yaml
    • Show in-toto attestation in CLI:

Retrieve provenance as JSON:

gcloud artifacts docker images describe $digest --show-provenance --format json > provenance.json

Verify provenance:

Using slsa-verifier

slsa-verifier verify-image $digest \
  --provenance-path provenance.json \
  --source-uri https://github.com/mchmarny/sds-demo \
  --builder-id https://cloudbuild.googleapis.com/GoogleHostedWorker@v0.3

Note, source-tag tag is not supported in GCB verification. The source-uri and builder-id come from materials.uri and builder.id respectively in the in-toto statement (provenance_summary.provenance.build.intotoStatement).

  • Extract the SBOM attestation (update the key location as needed)
cosign verify-attestation --type spdxjson \
  --key gcpkms://projects/cloudy-demos/locations/us-west1/keyRings/binauthz/cryptoKeys/binauthz-signer/cryptoKeyVersions/1 \
  $digest | jq -r '.payload' | base64 -d | jq .

Make sure the Cloud Build job completed before proceeding.

  • Navigate to Cloud Deploy pipelines
    • Drill into deploy-demo-pipeline
    • Show 2 environments (test, prod)
    • Drill into release, rollouts, and targets

  • Navigate to GKE workloads
    • Drill into hello (note non-gcp-built-image erred due to lack of attestation)
    • Navigate to exposed hello service at the bottom
    • Click on endpoint, and show version using /api/ping (should be same as the version in .version file in the repo)

  • Back to Cloud Deploy pipelines
    • Show promotion, review, and approval with manifest diffs and annotation comp (show more)

  • Navigate to Binary Authorization policy
    • Show cluster specific rules, and edit policy
    • Show Custom exemption rules
    • Show GKE cluster-specific rules

  • Show GKE Security Posture
    • Show concerns (note: there may not be much, if you are using brand new project for this demo)

Cleanup

setup/delete

Disclaimer

This is my personal project and it does not represent my employer. While I do my best to ensure that everything works, I take no responsibility for issues caused by this code.