demo-flux-multi-tenancy

Simple demo for managing multi-tenant clusters with Git and Flux v2.

Tools

fluxcd v2 >= 0.15 (installation guide)
minikube >= v1.18 (installation guide)

0. GitHub Token

Create a GitHub personal access token following this guide.

Remember to grant all repo permissions.

Once created save the token in an environment variable:

export GITHUB_TOKEN=<TOKEN_ID>
export GITHUB_USER=<MY_GITHUB_USER> (e.g. nikever)

1. Start minikube cluster

At the root level of this repository, execute:

export REPO_DIR=$(PWD)

Start a minikube cluster:

cd $REPO_DIR/minikube
make setup

By default, it will spin up a one node Kubernetes cluster of version v1.19.4 in a VirtualBox VM (CPUs=2, Memory=4096MB).

You can pass additional parameters to change the default values:

make setup cpu=4 memory=4096

Please referer to this Makefile for additional details on the cluster creation.

1. Bootstrap Flux V2

Bootstrap Flux via the cli:

flux bootstrap github \
    --owner $GITHUB_USER \
    --repository flux-fleet \
    --branch main \
    --path ./minikube-staging \
    --personal

The flux bootstrap command:

creates a repository if one doesn't exist
commits the manifests for the Flux components to the default branch at the specified path
installs the Flux components
configures the target cluster to synchronize with the specified path inside the repository

Wait for the pods to be up and running:

kubectl -n flux-system get pods -w

Clone the created repository:

mkdir $REPO_DIR/demo
cd $REPO_DIR/demo

git clone https://github.com/$GITHUB_USER/flux-fleet
cd flux-fleet

2. Deploy the sample Hello App via GitOps

Create a folder for the Hello App:

mkdir -p ./minikube-staging/hello-app/

Create a GitRepository manifest pointing to the sample Hello App repository's main branch:

flux create source git hello-app \
    --url https://github.com/nikever/kubernetes-hello-app \
    --branch main \
    --interval 1m \
    --export \
    > ./minikube-staging/hello-app/hello-app-source.yaml

Create a Flux Kustomization manifest. This configures Flux to build and apply the kustomize directory located in the Hello App repository under the manifests folder.

flux create kustomization hello-app \
  --source=hello-app \
  --path="./manifests" \
  --prune=true \
  --interval=1m \
  --export \
  > ./minikube-staging/hello-app/hello-app-kustomization.yaml

Commit and push to deploy in the cluster:

git add -A && git commit -m "Add Hello App"
git push

Wait for Flux to reconcile everything:

watch flux get sources git
watch flux get kustomizations

Wait for pods to be up and running:

kubectl get pods -n hello -w

Test the application:

curl -H "Host: hello-world.info" $(minikube ip)

Expected output:

Hello, world!
Version: 1.0.0
Hostname: hello-app-5c4957dcc4-l4mqz

3. Onboard tenant

Create folder for the team-a tenant:

mkdir -p ./minikube-staging/tenants/team-a

Generate the namespace, service account and role binding for the team-a:

flux create tenant team-a \
    --with-namespace=team-a \
    --export > ./minikube-staging/tenants/team-a/rbac.yaml

Create the sync manifests for the tenant Git repository:

flux create source git team-a \
    --namespace=team-a \
    --url=https://github.com/nikever/kubernetes-hello-app-tenant \
    --branch=main \
    --export > ./minikube-staging/tenants/team-a/flux.yaml

flux create kustomization team-a \
    --namespace=team-a \
    --service-account=team-a \
    --source=GitRepository/team-a \
    --path="staging/" \
    --export >> ./minikube-staging/tenants/team-a/flux.yaml

With the above configuration, the Flux instance running on the minikube-staging cluster will clone the team-a's repository, and it will reconcile the ./staging directory from the tenant's repo using the team-a service account. Since that service account is restricted to the team-a namespace, the team-a repository must contain Kubernetes objects scoped to the team-a namespace only.

Commit and push to deploy in the cluster:

git add -A && git commit -m "Onboard TeamA"
git push

Wait for Flux to reconcile everything:

watch flux get sources git -A
watch flux get kustomizations -A

Wait for pods to be up and running:

kubectl get pods -n team-a -w

Test the application:

curl -H "Host: team-a.staging.info" $(minikube ip)

Expected output:

Hello, world!
Version: 2.0.0
Hostname: hello-app-5c4957dcc4-l4mqz

4. Enforce policies with Kyverno

Copy the folder for infrastructure components:

cp -R $REPO_DIR/manifests/infrastructure minikube-staging/

Commit and push Kyverno:

git add minikube-staging/infrastructure/kyverno && git commit -m "Deploy Kyverno"
git push

Wait for Flux to reconcile everything:

watch flux get sources git -A
watch flux get kustomizations -A

Commit and push Kyverno-policiess:

git add minikube-staging/infrastructure && git commit -m "Deploy Kyverno policies"
git push

Wait for Flux to reconcile everything:

watch flux get sources git -A
watch flux get kustomizations -A

5. Clean up

To clean up:

Delete the minikube-staging cluster:

cd $REPO_DIR/minikube-staging
make delete

Delete the flux-fleet repository.

niccoloraspa/demo-flux-multi-tenancy