/homelab

My setup for a homelab kubernetes environment

Primary LanguageSmartyMIT LicenseMIT

What is this?

My setup for a homelab kubernetes environment with:

  • k3d for a single-host, multi-node cluster in Docker
  • k3sup for a multi-host, multi-node cluster on Raspberry Pis
  • Cloudflare for TLS, DNS, and proxying
  • nginx ingress with TLS termination using static Cloudflare origin certs and origin pull verification
  • cfsync for maintaining A record public IP entries for a NATed home network
  • minio for object storage
  • other apps of interest

Why build it?

A chance to review things I think I know. An opportunity to learn more. An itch to build. Boredom.

Why not use cert-manager and Let's Encrypt?

Saving on time and memory. Static origin certs are good enough for my purposes.

What manual steps did I take?

In Cloudflare:

  • Enable Full (strict) encryption mode
  • Generate a wildcard origin certificate for my domain and store in secrets.yaml
  • Download the Cloudflare CA for origin pull auth and store in values.yaml

In GitHub:

  • Create a GHCR_TOKEN secret with a personal access token having package write permission
  • Create the gh-pages orphan branch

To use the minio mc CLI:

  • Add a homelab alias to the ~/.mc/config.json file
  • Run make targets in the minio folder

To run on a single Raspberry Pi 3 (ARMv7):

  • Install k3d onto the Pi
  • Clone this project onto the Pi
  • Run make local-cluster
  • Copy the ~/.kube/config back to my main machine
  • Delete the default local-path StorageClass (probably should skip install and install custom)
  • Run make sync from the main machine

To set up a Raspberry Pi 4 (ARMv7) cluster:

  • Write empty ssh file in the microSD card boot partition
  • Write wpa_supplicant.conf to boot partition like:
country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="SSID"
    psk="PASSWORD"
}
  • SSH to pi@raspberrypi.local
  • Change pi user password
  • Add SSH pubkey to ~/.ssh/authorized_hosts
  • Run raspi-config to set hostname, lower GPU memory, expand root partition
  • Add cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory to /boot/cmdline.txt
  • Disable wifi power saving with sudo /sbin/iw wlan0 set power_save off and permanently in /etc/rc.local before the exit
  • Disable swap with sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove
  • Assign fixed IP
  • Repeat for all nodes
  • Install k3sup on my dev box
  • Run make cluster

How do I cut chart releases?

  1. Bump versions in cfsync/chart/Chart.yaml and helmfile.yaml.
  2. Push to main.
  3. Use the GitHub web UI to create a release with matching version tag.