My personal infrastructure.
Currently I use a self-hosted K3s instance on two VMs (one using aarch64
and one using x86_64
architecture; see Heterogeneous architecture below).
All resources are stored in this repository.
Since YAML is a nightmare, I use Jsonnet, which is a JSON superset with variables, functions etc.
To update cluster resources, I use kubecfg
.
nix-shell
- start an interactive shell with needed tools (it also creates ak
alias forkubectl
);kubecfg update *.jsonnet
- update Kubernetes resources.
Before above workflow became possible, I had to:
- point
example.com
to the cluster IP; - get a VM;
- install K3s (almost vanilla K3s, except
INSTALL_K3S_EXEC="server --disable traefik"
); - setup firewall;
- copy
/etc/rancher/k3s/k3s.conf
from the VM to~/.kube/config
on my workstation; - open or tunnel the Kubernetes API port (in my case
ssh -L 6443:localhost:6443 vm
is good enough); - allow cross-node traffic.
Also, by running kubectl apply -f vendor/
:
- install NGINX Ingress Controller and configure it as a default;
- install cert-manager.
I experiment with having nodes with different CPU architectures (currently x86_64
and aarch64
).
Since not every service is available on every architecture, I have tainted my nodes with necior/arch=x86_64:NoSchedule
xor necior/arch=aarch64:NoSchedule
and I have added appropriate tolerations to Deployments objects.