BigBang Template

This is a mirror of a government repo hosted on Repo1 by DoD Platform One. Please direct all code changes, issues and comments to https://repo1.dso.mil/platform-one/big-bang/customers/template**

[[TOC]]

This folder contains a template that you can replicate in your own Git repo to get started with Big Bang configuration. If you are new to Big Bang it is recommended you start with the Big Bang Quickstart before attempting customization.

The main benefits of this template include:

Isolation of the Big Bang product and your custom configuration
- Allows you to easily consume upstream Big Bang changes since you never change the product
- Big Bang product tags are explicitly referenced in your configuration, giving you control over upgrades
GitOps for your deployments configurations
- Single source of truth for the configurations deployed
- Historical tracking of changes made
- Allows tighter control of what is deployed to production (via merge requests)
- Enables use of CI/CD pipelines to test prior to deployment
- Avoids problem of helm upgrade using values.yaml that are not controlled
- Allows you to limit access to production Kubernetes cluster since all changes are made via Git
Shared configurations across deployments
- Common settings across deployments (e.g. dev, staging, prod) can be configured in one place
- Secrets (e.g. pull credentials) can be shared across deployments.
  
  NOTE: SOPS supports multiple keys for encrypting the same secret so that each environment can use a different SOPS key but share a secret.

Prerequisites

To deploy Big Bang, the following items are required:

Kubernetes cluster ready for Big Bang
A git repo for your configuration
Kubectl
GPG (Mac users need to read this important note)
SOPS
Git
Iron Bank Personal Access Token - Under your User Profile, copy the CLI secret.
Repo1 Personal Access Token - You will need read_repository permissions.
Helm
Kustomize

In addition, the following items are recommended to assist with troubleshooting:

Setup

This template supports a multi-environment template for two distinct deployments: prod and dev. Additional environments can be added by replicating one of the existing folders.

Each environment consists of a Kubernetes manifest containing Flux resources (bigbang.yaml), a Kustomization file (kustomization.yaml), values to pass to Big Bang (configmap.yaml), secrets (secrets.enc.yaml), and additional files used to deploy resources. All of the environments share a base folder to allow reusability of values between environments.

To insure variables (e.g. ${fp}) are set correctly, execute all of the steps below in the same terminal window.

Create Git Repository

We need to work off our own Git repo for storing configuration. So, you should fork this repo into a private Git repo owned by yourself or your project. Then, clone your repo locally.

git clone https://<your domain>/<your repo>.git
cd <your repo>

# Create branch for your changes
git checkout -b template-demo

It is recommended that you create your own branch so that you can pull the original repository's main branch as a mirror to keep it in sync.

Create GPG Encryption Key

To make sure your pull secrets are not compromised when uploaded to Git, you must generate your own encryption key:

Keys should be created without a passphrase so that Flux can use the private key to decrypt secrets in the Big Bang cluster.

# Generate a GPG master key
# The GPG key fingerprint will be stored in the $fp variable
export fp=`gpg --quick-generate-key bigbang-sops rsa4096 encr | sed -e 's/ *//;2q;d;'`
gpg --quick-add-key ${fp} rsa4096 encr

# By default our key is set to expire in 2 years.
# Here we reduce this down to 14 days for learning/demo purposes
gpg --quick-set-expire ${fp} 14d

# Rekey the .sops.yaml
# This ensures your secrets are only decryptable by your key

## On linux
sed -i "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

## On MacOS
sed -i "" "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

# Save encrypted secrets into Git
# Configuration changes must be stored in Git to take affect
git add .sops.yaml
git commit -m "chore: update default encryption key"
git push --set-upstream origin template-demo

Add TLS Certificates

The base/configmap.yaml is setup to use the domain bigbang.dev by default. A demo TLS certificate is provided in base/bigbang-dev-cert.yaml to use. Certificates should be encrypted before pushing to Git since they contain both the public and private key.

cd base

# Encrypt the existing certificate
sops -e bigbang-dev-cert.yaml > secrets.enc.yaml

# Save encrypted TLS certificate into Git
git add secrets.enc.yaml
git commit -m "chore: add bigbang.dev tls certificates"
git push

Add Pull Credentials

You will need pull credentials for Iron Bank to retrieve images for Big Bang.

Secrets can be specific to an environment if they are located in that environment's folder (e.g. prod, dev). Or, they can be shared between environments if located in the base directory.

# Edit the same secret holding your TLS certificates to add the pull credentials
sops secrets.enc.yaml

Add the following contents to the newly created sops secret. Put your Iron Bank user/PAT where it states replace-with-your-iron-bank-user and replace-with-your-iron-bank-personal-access-token.

The name of the secret must be common-bb if the secret is in the base folder or environment-bb if the secret is in the dev or prod folder. The environment-bb values take precedence over the common-bb values.

apiVersion: v1
kind: Secret
metadata:
   name: common-bb
stringData:
   values.yaml: |-
      registryCredentials:
      - registry: registry1.dso.mil
        username: replace-with-your-iron-bank-user
        password: replace-with-your-iron-bank-personal-access-token
      istio:
        # Leave the TLS certificate info here

When you save the file, it will automatically re-encrypt your secret using SOPS.

# Save pull credentials into Git
git add secrets.enc.yaml
git commit -m "chore: added iron bank pull credentials"
git push

Your private key to decrypt these secrets is stored in your GPG key ring. You must NEVER export this key and commit it to your Git repository since this would compromise your secrets.

Configure for GitOps

We need to reference your git repository so that Big Bang will use the configuration. Add your repository into the GitRepository resource in dev/bigbang.yaml:

cd ../dev/

Replace your forked Git repo where it states replace-with-your-git-repo. Replace replace-with-your-branch with your branch name (e.g. template-demo as created above).

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
   name: environment-repo
   namespace: bigbang
spec:
   interval: 1m
   url: https://replace-with-your-git-repo.git
   ref:
     branch: replace-with-your-branch
   secretRef:
     name: private-git

The Kustomization resource contains the path in your repo to the kustomization.yaml to start with. If your folder changes, makes sure to update spec.path with the new path.

Now, save and commit your change:

git add bigbang.yaml
git commit -m "chore: updated git repo"
git push

Deploy

Big Bang follows a GitOps approach to deployment. All configuration changes will be pulled and reconciled with what is stored in the Git repository. The only exception to this is the initial manifests (e.g. bigbang.yaml) which points to the Git repository and path to start from.

Deploy SOPS private key for Big Bang to decrypt secrets

# The private key is not stored in Git (and should NEVER be stored there).  We deploy it manually by exporting the key into a secret.
kubectl create namespace bigbang
gpg --export-secret-key --armor ${fp} | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey.asc=/dev/stdin

Create imagePullSecrets for Flux

# Image pull secrets for Iron Bank are required to install flux.  After that, it uses the pull credentials we installed above
kubectl create namespace flux-system

# Adding a space before this command keeps our PAT out of our history
 kubectl create secret docker-registry private-registry --docker-server=registry1.dso.mil --docker-username=<Your IronBank Username> --docker-password=<Your IronBank Personal Access Token> -n flux-system

Create Git credentials for Flux

# Flux needs the Git credentials to access your Git repository holding your environment
# Adding a space before this command keeps our PAT out of our history
 kubectl create secret generic private-git --from-literal=username=<Your Repo1 Username> --from-literal=password=<Your Repo1 Personal Access Token> -n bigbang

Deploy Flux to handle syncing

# Flux is used to sync Git with the the cluster configuration
# If you are using a different version of Big Bang, make sure to update the `?ref=1.29.0` to the correct tag or branch.
kustomize build https://repo1.dso.mil/platform-one/big-bang/bigbang.git//base/flux?ref=1.29.0 | kubectl apply -f -

# Wait for flux to complete
kubectl get deploy -o name -n flux-system | xargs -n1 -t kubectl rollout status -n flux-system

Deploy Big Bang

kubectl apply -f bigbang.yaml

# Verify 'bigbang' namespace is created
kubectl get namespaces

# Verify Pull from Git was successful
kubectl get gitrepositories -A

# Verify Kustomization was successful
# NOTE: The Kustomization resource may fail at first with an error about the istio-system namespace.  This is normal since the Helm Release for istio will create that namespace and it has not run yet.  This should resolve itself within a few minutes
kubectl get -n bigbang kustomizations

# Verify secrets and configmaps are deployed
# At a minimum, you will have the following:
#  secrets: sops-gpg, private-git, common-bb, and environment-bb
#  configmaps: common, environment
kubectl get -n bigbang secrets,configmaps

# Watch deployment
watch kubectl get hr,po -A

# Test deployment by opening a browser to "kiali.bigbang.dev" to get to the Kiali application deployed by Istio.
# Note that the owner of "bigbang.dev" has setup the domain to point to 127.0.0.1 for this type of testing.
# If you are deployed on a remote host you will need to point "kiali.bigbang.dev" to your cluster master node via your /etc/hosts file

If you cannot get to the main page of Kiali, it may be due to an expired certificate. Check the expiration of the certificate in base/configmap.yaml.

For troubleshooting deployment problems, refer to the Big Bang documentation.

You now have successfully deployed Big Bang. Your next step is to customize the configuration.

Customize

Enable a package

In dev/configmap.yaml, enable Twistlock
```
twistlock:
  enabled: true
```

Push changes to Git

git add configmap.yaml
git commit -m "feat: enable twistlock"
git push

Big Bang will automatically pick up your change and make the necessary changes.

# Watch deployment for twistlock to be deployed
watch kubectl get hr,po -A

# Test deployment by opening a browser to "twistlock.bigbang.dev" to get to the Twistlock application

Update the Big Bang Version

To minimize the risk of an unexpected deployment of a BigBang release, the BigBang release version is explicitly stored in the kustomization.yaml files and can be updated for a planned upgrades. The default release is stored in base/kustomization.yaml, but can be overridden in a specific environment like dev/kustomization.yaml.

Reference for the Big Bang kustomize base:

bases:
- https://repo1.dso.mil/platform-one/big-bang/bigbang.git/base/?ref=1.29.0

Reference for the Big Bang helm release:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
   name: bigbang
spec:
   ref:
      $patch: replace
      tag: "1.29.0"

To update dev/kustomization.yaml, you would create a mergePatch like the following:

patchesStrategicMerge:
- |-
  apiVersion: source.toolkit.fluxcd.io/v1beta1
  kind: GitRepository
  metadata:
    name: bigbang
  spec:
    interval: 1m
    ref:
      $patch: replace
      tag: "1.29.0"

This does not update the kustomize base, but it is unusual for that to change.

Then, commit your change:

   git add kustomization.yaml
   git commit -m "feat(dev): update bigbang to 1.29.0"
   git push

It may take Big Bang up to 10 minutes to recognize your changes and start to deploy them. This is based on the interval set for polling. You can force Big Bang to recheck by running the sync.sh script.

It is recommended that you track Big Bang releases using the version. However, you can use branch in place of tag if needed. The kustomize base uses Go-Getter's syntax for the reference. The helm release (GitRepository) resource uses the GitRepository CRD's syntax.

When you are done testing, you can update the reference in base (and delete this setting in dev) to update Big Bang in all environments.

Do not forget to also update the base/kustomization.yaml's base: reference to point to the new release.

Update the domain

Big Bang deploys applications to *.bigbang.dev by default. You can override the bigbang.dev domain to your domain by updating base/configmap.yaml and adding the following:

domain: insert-your-domain-here

In addition, you will need to update the TLS certificates by updating base/secrets.enc.yaml.

# Open and edit the encrypted file
sops base/secrets.enc.yaml

After saving the secrets.enc.yaml file, it will be automatically re-encrypted.

# Push changes to Git
git add base/configmap.yaml base/secrets.enc.yaml
git commit -m "chore: updated domain and tls certificates"
git push

If you have different certificates for dev and prod, you can also put the values in dev/secrets.enc.yaml or prod/secrets.enc.yaml respectively. The name of the secret must be common-bb if the secret is in the base folder or environment-bb if the secret is in the dev or prod folder. The environment-bb values take precedence over the common-bb values. Make sure to add the file to kustomization.yaml as a resource if it is not already.

Use NodePorts for Istio Ingress Gateway

Istio Ingress Gateway is deployed with the LoadBalancer service type by default. You can configure Istio to use nodePorts instead by updating dev/configmap.yaml:

istio:
  ingressGateways:
    public-ingressgateway:
      type: "NodePort"
      nodePortBase: 30000

Node ports are assigned starting from nodePortBase. The nodePortBase specifies the start of a range of 4 unused node ports. Node port will be assigned as follows: Port 15021 (Status) = nodePortBase, Port 80 = nodePortBase+1, Port 443 = nodePortBase+2, Port 15443 (SNI) = nodePortBase+3. Node port base should be in the range from 30000 to 32764. Alternatively, the kubernetesResourceSpec can be used to configure all port parameters. If nodePortBase isn't specified ports will be assigned randomly.

Additional Big Bang values

For additional configuration options, refer to the Big Bang and Big Bang Package documentation. Big Bang values can be passed down in the configmap.yaml or secrets.enc.yaml. See the Kubernetes documentation on configmaps and secrets for differences between the two. Secrets should always be SOPS encrypted before committing to Git.

NOTE: The dev template includes several overrides in the configmap.yaml to minimize resource usage and increase polling time in a development environment. They are provided for convenience and are NOT required.

Additional resources

Using Kustomize, you can add additional resources to the deployment if needed. Read the Kustomization documentation for further details.

Secrets

You have already created a GPG Encryption Key pair and deployed encrypted pull credentials above. Here are some additional scenarios you may encounter with secrets.

Key Vault

SOPS supports several key vaults to help control access to your secure keys including:

You will need to update .sops.yaml with your configuration based on the links above.

Key Rotation

If you need to rotate your GPG encryption keys for any reason, you will also need to re-encrypt any encrypted secrets.

Update .sops.yaml configuration file .sops.yaml holds all of the key fingerprints used for SOPS. Update pgp's value to the new key's fingerprint. You can list your locally stored fingerprints using gpg -k.
```
creation_rules:
- encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_NEW_KEY_FINGERPRINT_HERE
```
Re-encrypt secrets.enc.yaml with your new SOPS keys. This will decrypt the file with the old key and re-encrypt with the new key.
```
sops updatekeys base/secrets.enc.yaml -y
# Repeat this for all other encrypted files (e.g. dev/secrets.enc.yaml)
```

Deploy new SOPS private key for Big Bang to decrypt secrets

# The private key is not stored in Git (and should NEVER be stored there).  We deploy it manually by exporting the key into a secret.
kubectl delete secret sops-gpg -n bigbang
gpg --export-secret-key --armor INSERT_NEW_KEY_FINGERPRINT_HERE | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey.asc=/dev/stdin

Commit changes

   git add .sops.yaml **/secrets.enc.yaml
   git commit -m "chore: rekey secrets"
   git push

Multiple Keys

You can encrypt files with SOPS using more than one key to allow different keys to decrypt the same file. The encrypted file contains copies of the data encrypted with each key and all of the public keys needed to re-encrypt the file if changes are made.

Only one of the private keys is required to decrypt the file

Add the second key's fingerprint to .sops.yaml:

creation_rules:
- encrypted_regex: '^(data|stringData)$'
  pgp: ORIGINAL_KEY
      ,INSERT_SECOND_KEY_HERE

Re-encrypt all encrypted files with your new SOPS keys. This will decrypt the file with the original key and re-encrypt with both of the keys.
```
sops updatekeys base/secrets.enc.yaml -y
# Repeat this for all other encrypted files (e.g. dev/secrets.enc.yaml)
```

Commit changes

   git add .sops.yaml **/secrets.enc.yaml
   git commit -m "chore: added second key to secrets"
   git push

Different keys for different environments

In our template, we have a dev and a prod environment with a shared base. Let's say we wanted the following:

Shared Iron Bank pull credential
Different database passwords for dev and prod
Different SOPS keys for dev and prod

Setup .sops.yaml for multiple folders:

creation_rules:
# Base is shared, so add fingerprints of both keys
- path_regex: base/.*
  encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_DEV_KEY_FINGERPRINT_HERE
    ,INSERT_PROD_KEY_FINGERPRINT_HERE
- path_regex: dev/.*
  encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_DEV_KEY_FINGERPRINT_HERE
- path_regex: prod/.*
  encrypted_regex: '^(data|stringData)$'
  pgp: INSERT_PROD_KEY_FINGERPRINT_HERE

Re-encrypt all encrypted files with your SOPS keys. This will decrypt the file with the original private key and re-encrypt with the new keys according to your path_regex.

If you do not have secrets.enc.yaml in dev or prod, you can can copy the one in base to test out these commands.

sops updatekeys base/secrets.enc.yaml -y
sops updatekeys dev/secrets.enc.yaml -y
sops updatekeys prod/secrets.enc.yaml -y

There is an excellent tutorial on multiple key SOPS here.

Commit changes

   git add .sops.yaml **/secrets.enc.yaml
   git commit -m "chore: split dev and prod keys"
   git push

Modifying an encrypted file

Updating values in an encrypted file can be achieved by simply opening the file with sops:

sops base/secrets.enc.yaml

When you save the file, sops automatically re-encrypts it for all of the keys specified in .sops.yaml.

Multi-environment Workflow

In this template, we have a dev and prod environment. Your specific situation deployment may have more. Our intended workflow is:

Test changes in the dev environment before deploying into prod
Keep dev as close as possible to prod by sharing values
Maintain dev and prod specific settings for resources, external connections, and secrets

To start, we may have the following in each folder:

base
- Iron Bank pull credentials
- Big Bang release reference
- Application settings
dev
- Dev domain name
- Dev TLS certificates
- Minimized resource values (e.g. memory, cpu)
- Dev external connections and credentials
prod
- Prod domain name
- Prod TLS certificates
- Prod external connections and credentials

Big Bang dev value changes can be made by simply modifying dev/configmap.yaml. base and dev create two separate configmaps, named common and environment respectively, with the environment values taking precedence over common values in Big Bang.

The same concept applies to dev secret changes, with two separate secrets named common-bb and environment-bb used for values to Big Bang, with the environment-bb values taking precedence over the common-bb values in Big Bang.

If a new resource must be deployed, for example a TLS cert, you must add a resources: section to the kustomization.yaml to refer to the new file. See the base directory for an example.

wtomaz808/big-bang-template

BigBang Template

Prerequisites

Setup

Create Git Repository

Create GPG Encryption Key

Add TLS Certificates

Add Pull Credentials

Configure for GitOps

Deploy

Customize

Enable a package

Update the Big Bang Version

Update the domain

Use NodePorts for Istio Ingress Gateway

Additional Big Bang values

Additional resources

Secrets

Key Vault

Key Rotation

Multiple Keys

Different keys for different environments

Modifying an encrypted file

Multi-environment Workflow