This repository is my home Kubernetes cluster in a declarative state. Flux watches my cluster directory and makes the changes to my cluster based on the YAML manifests.
Feel free to open a Github issue if you have any questions.
This repository is built off the k8s-at-home/template-cluster-k3s repository.
- Talos: Built using talhelper
- fluxcd: Sync kubernetes cluster with this repository.
- SOPS: Encrypts secrets which is safe to store - even to a public repository.
- Cilium: For internal cluster networking, also as load balancer to expose services.
- cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
- ingress-nginx: Ingress controller for services.
- authelia: Full featured authentication server.
- rook-ceph: Cloud native distributed block storage for Kubernetes
- nfs-subdir-external-provisioner: Provides persistent volumes from NFS server.
- Intel GPU plugin: Access intel gpu available on nodes.
- node-feature-discovery: Discover features available on nodes.
- Prometheus: Scraping metrics from the entire cluster
- Grafana: Visualization for the metrics from Prometheus
The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.
./cluster
├── ./base # entrypoint to Flux
└── ./apps # everything is here
Inside the apps directory, I divided all the apps using their namespaces. Every app will have its own "Fluxtomization" file that describe their manifests and dependencies.
Incoming http and https traffics from outside of my network are forwarded from OPNSense firewall into ingress-nginx pod with a LoadBalancer service using MetalLB layer2 implementation.
So, basically this is how the http(s) traffic flows:
Internet -> OPNSense firewall -> ingress-nginx service -> Kubernetes pod
Ingress-nginx service is using Local externalTrafficPolicy so I can track the real IP of clients trying to access my services.
For important backend services like my OPNSense, I use nginx.ingress.kubernetes.io/whitelist-source-range annotation to only allow access from my internal networks.
My certificates are managed with cert-manager using LetsEncrypt as the CA.
Secrets are encrypted using sops before being pushed into this repository. The encrypted secrets are then decrypted by sops using the private key inside the cluster. For encryption/decryption, I use age. The public key to encrypt the secret is in .sops.yaml. Secrets environment variables for the cluster are in cluster-secret-vars.yaml. The non secret variables are in cluster-vars.yaml.
Metrics scraping for the cluster are done using Prometheus.
Dashboards included in my cluster are:
- The provided dashboard from kube-prometheus-stack
- Ingress-nginx dashboard from here
- Fluxcd dashboard from here
- Rook-ceph dashboards from here
To add your own dashboard, create a configmap with the data include the json file of the dashboard and add label grafana_dashboard: "1" to the manifest.
The sidecar container from this image will mount the dashboard into your grafana pod.
A lot of inspiration for my cluster came from this awesome template
- Use redis operator