Home k8s infrastructure

Deploying a cluster with Ansible and Terraform backed by ArgoCD and SOPS.

Overview

🧱 Components

cert-manager - SSL certificates - with Cloudflare DNS challenge
flannel - CNI (container network interface)
ArgoCD - GitOps tool for deploying manifests from the cluster directory
hajimari - start page with ingress discovery
rook.io - ceph storage class for k8s
nfs - used for cold storage on QNAP
metallb - bare metal load balancer
traefik - ingress controller

For provisioning the following tools are used:

Ubuntu - this is a pretty universal operating system that supports running all kinds of home related workloads in Kubernetes
Ansible - this is used to provision the Ubuntu operating system to be ready for Kubernetes and also to install k8s
Terraform - in order to help with the DNS settings this is used to provision an already existing Cloudflare domain and DNS settings

📝 Setup

💻 Systems

Nodes running Ubuntu Server 20.04. These nodes are bare metals.
A Cloudflare account with a domain, this will be managed by Terraform.
QNAP used as NFS, S3 and Backup storage.

🧠 Devcontainer

For fast setup I use devcontainer to have same environment across different devices. See more inside .devcontainer and at Devcontainers

🔧 Tools

Install the most recent versions of the following command-line tools on your workstation, if you are using Homebrew on macOS or Linux skip to steps 3 and 4.
- Required: age, ansible, go-task, ipcalc, jq, kubectl, pre-commit, sops, terraform, yq, argocd CLI
- Recommended: direnv, helm, kustomize, prettier, stern, yamllint
This guide heavily relies on go-task as a framework for setting things up. It is advised to learn and understand the commands it is running under the hood.
Install go-task via Brew
```
brew install go-task/tap/go-task
```
Install workstation dependencies via Brew
```
task init
```

⚠️ pre-commit

It is advisable to install pre-commit and the pre-commit hooks that come with this repository. sops-pre-commit will check to make sure you are not by accident committing un-encrypted secrets.

Enable Pre-Commit
```
task precommit:init
```
Update Pre-Commit, though it will occasionally make mistakes, so verify its results.
```
task precommit:update
```

📂 Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Argo CD will apply them.

core main folder for argocd deployment and cluster management. In projects folder there are all deployed applications definitions used by Argo CD
apps directory (depends on core) is where your common applications (grouped by namespace) could be placed
system place for system applications like ingress or oauth/identity etc.

📁 cluster
├──📁 apps
│   ├──📁 default
│   ├──📁 networking
│   └──📁 system-upgrade
├──📁 core
│   └──📁 argocd
│       ├──📁 base
│       └──📁 projects
└──📁 system

🚀 Deployment

🔐 Setting up Age

I assume you already have generated age key pair to be used otherwise you need to generate one. Export the SOPS_AGE_KEY_FILE variable in your bashrc, zshrc or config.fish and source it, e.g.

export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
source ~/.bashrc

☁️ Global Cloudflare API Token

In order to use Terraform and cert-manager with the Cloudflare DNS challenge you will need to create a API Token.

Head over to Cloudflare and create a API Token by going here.
Under the API Tokens section, create a scoped API Token.
Use the API Token in provision/terraform/cloudflare and cluster/system/cert-manager.

⚡ Preparing Ubuntu with Ansible

📍 Nodes are not security hardened by default, you can do this with dev-sec/ansible-collection-hardening or something similar.

Ensure you are able to SSH into you nodes from your workstation with using your private ssh key. This is how Ansible is able to connect to your remote nodes.
Install the deps by running task ansible:deps
Verify Ansible can view your config by running task ansible:list
Verify Ansible can ping your nodes by running task ansible:adhoc:ping
Finally, run the Ubuntu Prepare playbook by running task ansible:playbook:ubuntu-prepare
If everything goes as planned you should see Ansible running the Ubuntu Prepare Playbook against your nodes.

⛵ Installing k8s with Ansible

This section is under works and it's still not possible to prepare k8s using ansible.

☁️ Configuring Cloudflare DNS with Terraform

📍 Review the Terraform scripts under ./provision/terraform/cloudflare/ and make sure you understand what it's doing (no really review it). If your domain already has existing DNS records be sure to export those DNS settings before you continue. Ideally you can update the terraform script to manage DNS for all records if you so choose to.

Pull in the Terraform deps by running task terraform:init:cloudflare
Review the changes Terraform will make to your Cloudflare domain by running task terraform:plan:cloudflare
Finally have Terraform execute the task by running task terraform:apply:cloudflare

If Terraform was ran successfully you can log into Cloudflare and validate the DNS records are present.

🐙 GitOps with ArgoCD

📍 Here we will be installing ArgoCD after some quick bootstrap steps.

Verify ArgoCD can be installed
```
argocd version
# argocd: v2.3.1
# ...
```

Pre-create the argocd namespace

kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -

Add the Age key in-order for ArgoCD to decrypt SOPS secrets

cat $SOPS_AGE_KEY_FILE |
    kubectl -n argocd create secret generic sops-age \
    --from-file=age.agekey=/dev/stdin

Verify all files ending with *.sops.yaml or *.sec.yaml are encrypted with SOPS

Push you changes to git

git add -A
git commit -m "encrypting secrets"
git push

Install Argo CD

kubectl apply -k ./cluster/core/argocd/base

Verify Argo CD components are running in the cluster
```
kubectl get pods -n argocd
```
If all goes well and you have port forwarded 80 and 443 in your router to the METALLB_TRAEFIK_ADDR IP, in a few moments head over to your browser and you should be able to access https://hajimari.CLOUDFLARE_DOMAIN

🎉 Congratulations you have a Kubernetes cluster managed by Argo CD, your Git repository is driving the state of your cluster.

📣 Post installation

👉 Cluster maintenance

This section will be about upgrading k8s and onther components on your cluster using Ansible.

chrismgonzalez/home-ops-1