Build a Kubernetes (k3s) cluster with RPis and utilize GitOps for managing cluster state. I would like to give a shout-out to k8s-gitops, the big brother of this repo, created by @billimek.
This repo uses a lot of multi-arch images provided by raspbernetes/multi-arch-images.
Note: A lot of files in this project have @CHANGEME comments, these are things that are specific to my set up that you may need to change.
- 3x Raspberry Pi 4 (recommended 4GB RAM model)
- 3x SD cards (recommended 32GB)
- 3x USB 3.x flash drives (recommended for local storage)
- A NFS server for storing persistent data (recommended for shared storage)
Note: I use the fish shell for a lot of my commands. Some will work in Bash but others will not, see here for more information.
.
├── ./ansible # Ansible playbook to run after the RPis have been flashed
├── ./deployments # Flux will only scan and deploy from this directory
├── ./setup # Setup of the cluster
├── ./secrets # Scripts to generate secrets for Sealed Secrets
└── ./docs # Documentation
| IP | Function |
|---|---|
| 192.168.1.1 | Router (USG) |
| 192.168.1.170 | NFS Server |
| 192.168.42.1/24 | k3s cluster CIDR, VLAN 42 |
| 192.168.42.23 | k3s master (k3s-master) |
| 192.168.42.24 | k3s worker (k3s-worker-a) |
| 192.168.42.25 | k3s worker (k3s-worker-b) |
See ubuntu.md
Ansible is a great automation tool and here I am using it to provision the RPis.
See ansible.md and review the files in the ansible folder.
k3sup is a neat tool provided by @alexellis that helps get your k3s cluster up and running quick.
For manual deployment see k3sup.md, and for an automated script see bootstrap-cluster.sh
Helm is a package manager for Kubernetes.
Flux is the GitOps tool I've chosen to have this Git Repository manage my clusters state.
For manual deployment see helm-flux.md, and for an automated script see bootstrap-cluster.sh
Sealed Secrets are a "one-way" encrypted Secret that can be created by anyone, but can only be decrypted by the controller running in the target cluster. The Sealed Secret is safe to share publicly, upload to git repositories, give to the NSA, etc. Once the Sealed Secret is safely uploaded to the target Kubernetes cluster, the sealed secrets controller will decrypt it and recover the original Secret.
See sealed-secrets.md and review the files in the secrets folder.
MetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.
Review the file metallb.yaml
Cert-Manager will automatically provision and manage TLS certificates in Kubernetes. In this setup I am using Cloudflare as the DNS challenge.
NGINX Ingress Controller is an Ingress controller that uses ConfigMap to store the NGINX configuration.
Review the file nginx-ingress.yaml