Homelab
Don't be fooled, having a home server is really just hundreds of hours of badblocks.
Features
- Lots of self-hosted services
- Flux GitOps with this repository (cluster directory)
- Ansible node provisioning and K3s setup (Ansible roles and playbooks)
- Terraform DNS records (terraform)
- SOPS secrets stored in Git
- Renovate bot dependency updates
- WireGuard VPN pod gateway via paid service
- WireGuard VPN proxy hosted on VPS
- Cloudflared HTTP tunnel
- K8s gateway for local DNS resolution to cluster and NGINX ingress controller
- Both internal & external services with a service gateway
- OIDC authentication with LDAP
- Automatic Cloudflare DNS updates (ddns cronjob)
- MetalLB bare metal K8s network loadbalancing
- Calico CNI
- ZFS
- JBOD mergerfs union NFS with SnapRAID backup for low-touch media files (snapraid-runner cluster cronjob)
- Restic backups to remote and local buckets (backup namespace)
- go-task shorthand for useful commands (Taskfile and taskfiles)
Usage
Setup and usage is inspired heavily by this homelab gitops template and the k8s-at-home community. You can find similar setups with the k8s at home search. Historical revisions of this repository had rootless Podman containers deployed with ansible as systemd units.
Setup
task init
Then, provision your infrastructure.
task ansible:{list,setup,kubernetes,status}
Edit provision/terraform/cloudflare/secret.sops.yaml
with your own values and encrypt with task sops:encrypt -- <filepath>
.
Setup Cloudflare DNS.
task terraform:{init,cloudflare-plan,cloudflare-apply}
Deploy
Kubernetes
Verify flux can be installed. Then, push changes to remote repo and install.
task cluster:{verify,install}
Push latest to repo - you can use the wip.sh script for that with task wip
.
task cluster:{reconcile,resources}
Bastion server
Edit provision/terraform/bastion/secret.sops.yaml
with your own values. Generate WireGuard keys.
Deploy the remote bastion VPN server.
task terraform:{init,plan,apply}
Then, setup VPN services.
task ansible:bastion
Deployments
Most deployments in this repo use an app-template
chart with these configuration options.
Update
The Renovate bot will help find updates for charts and images. Install Renovate Bot, add to your repository and view Renovate bot activity, or use the self-hosted option.
Hardware
Resources
Memory
- Why use ECC (discussion)
- If you love your data, use ECC RAM.
- Error rates increase rapidly with rising altitude.
Storage
Controller
I used a widely-known and inexpensive method to add additional SATA storage via a Host Bus Adapter (HBA). I purchased a Dell Perc H310 a long while back. Mine did come from overseas, but it turned out to be legit. This video shows how it can be flashed to an LSI 9211-8i IT (see also 1, 2).
Here are other recommended controllers.
2.5" drive stackers
These printable stackers are great for stacking SSDs in a homelab.
Home automation
Zigbee/Z-Wave
- USB Zigbee/Z-Wave receiver and upgrade Zigbee firmware for compatibility with Home Assistant (notice this issue)
Software
Linux
SSH
Here's a nice convenience for setting up authorized_keys
stored on Github or Gitlab:
curl https://github.com/<username>.keys -o authorized_keys
You could pipe the output to sed
to only grab a specific line sed '4!d'
.
Check disks
Here's a handy script to automatically test disks with badblocks and SMART: Spearfoot/disk-burnin-and-testing.
Testing disks takes a long time for larger drives, but it's worth it to be thorough before determining whether to make a return. This is a destructive test, so it's probably best to use /dev/disk/by-id
to be certain you're targeting the correct drive.
Use tune2fs -l <partition>
to identify the block size.
sudo badblocks -wsv -b 4096 /dev/sda > sda_badblocks.txt
Here's some additional advice from /r/DataHoarders.
JBOD
MergerFS is a union filesystem for pooling drives together. It's a great pair with SnapRAID. An alternative is SnapRAID-BTRFS.
mkdir /mnt/disk{1,2,3,4}
mkdir /mnt/parity1 # adjust this command based on your parity setup
mkdir /mnt/storage # this will be the main mergerfs mount point (a collection of your drives)
Mount drives to these folders, then add /etc/fstab
entries by ID.
ls /dev/disk/by-id
You must also include an entry for the MergerFS union, such as:
/mnt/disk* /mnt/storage fuse.mergerfs allow_other,use_ino,cache.files=partial,dropcacheonclose=true,category.create=mfs,fsname=mergerfs,minfreespace=10G 0 0
See also perfectmediaserver: MergerFS
Remember, for data that's irreplaceable RAID is not a backup.
ZFS
Install zfs-dkms
and zfs-utils
, and be sure to have linux-headers
installed for dkms to work. Update the ZFS libraries together using a AUR helper.
OS Installation
Use Ventoy to bundle bootable ISO and IMG images on a single USB.
Media
For a media server, it's a good idea to understand digital video.
Troubleshooting
Network
Debug DNS issues
kubectl run curl --rm=true --stdin=true --tty=true --restart=Never --image=docker.io/curlimages/curl --command -- /bin/sh -
curl -k https://kubernetes:443; echo
Ensure you're using iptables-legacy. See also nftables.
iptables --version
# iptables v1.8.7 (legacy)
Flush the iptables in between installs. Also check the CNI installation for issues (such as configuration for hardware with multiple NICs).