Bare metal Kubernetes using k3s

Instead of virtual machines (VMs), deploy a production-grade Kubernetes cluster directly on bare metal servers. We will use k3s along with k3sup. K3s is a lightweight Kubernetes distribution and probably easiest and fastest way to setup a Kubernetes cluster in HA mode.

Install Prerequisite:

Install k3sup on your machine. You can use the installer on MacOS and Linux, or visit the Releases page to download the executable for Windows.
Servers have been already created and initialized using a Ubuntu (22.04/20.04) or Debian (12/11) server operating system. Everything have been tested using Ubuntu (22.04).
Make sure server host names are unique, otherwise K3s clustering process will fail. If not go to nano /etc/hostname and change each node to have unique host name.

Minimum node requirements:

CPU: 4c/8t or more
RAM: 16GB or more
Disk: 256GB SSD/NVMe or more

For high-available setup:

Minimum 2 or more nodes
Network connectivity between nodes

Network connectivity between nodes can be private or public although private connectivity is recommended. Please see high-level UFW firewall rules in bash-templates/ufw-cluster.sh.

Step-by-step

Add files into `data` folder

See example files for your reference.

cluster-ips.txt [Required] - List of IPs for k3s cluster nodes. Each IP on a new line.
cluster-user.txt [Required] - Cluster user with sudo permission to be used for k3s installation via SSH. It assumes that authorization keys for SSH is already uploaded to host server.
cf-token.env [Required] - For DNS01 challenge to issue Let's Encrypt Certificates for DNS zone hosted by Cloudflare DNS.
google-oidc-config.env [Required] - For Traefik Forward Auth to protect various dashboards including Traefik, Grafana, etc.
google-oidc-secrets.env [Required] - For Traefik Forward Auth to protect various dashboards including Traefik, Grafana, etc.

Update k3s-chart values

Rename example.values.yaml to values.yaml and modify values according to your requirements. Basically this files needs a list of domain names and ACME contact email to provision Let's Encrypt Certificates.

Generate user data file

Execute generate-init.sh (Optional) and generate-k3s.sh. This will create a set of files to configure hosts, bootstrap Kubernetes cluster using K3s, apply some basic manifests and helm charts to newly created cluster.

bash generate-init.sh
bash generate-k3s.sh

Output from `generate-init.sh`

These files need to be executed remotely on the host as sudo user.

init.sh to apply additional configuration to hosts including firewall changes and install a few apt packages. You can copy this file to host servers and execute.
cloudflared.sh [Optional] to install remotely managed Cloudflare tunnel on each host to securely SSH and access Kubernetes control plane behind the firewall.
cloudflared-sshd.sh [Optional] to setup SSH using Cloudflare short-lived certificate which can be used to render the SSH console in the browser.

To enabled Cloudflare tunnel,

Output from `generate-k3s.sh`

Execute k3s.sh to install K3S. Script takes local SSH key (e.g. $HOME/.ssh/id_ed25519) for provisioning the cluster. After cluster is created script will save the kubeconfig in root directory.
Execute kubectl.sh to apply various Kubernetes manifest files and create resources
- to enable observability on cluster
- to expose dashboards for Traefik, Grafana, Prometheus.
- to add add Traefik Forward Auth to protect dashboards behind Google Authentication

Location of kubeconfig

kubeconfig will be saved to directory where you ran k3s.sh.

export KUBECONFIG=/Users/../../k3s/kubeconfig
kubectl config use-context default
kubectl get node -o wide

abhishektiwari/bare-metal-k3s