Runbook

Requirements

Install Ansible : https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
Install k3sup : https://github.com/alexellis/k3sup
Install kubectx & kubens : https://github.com/ahmetb/kubectx
Install Helm : https://helm.sh/docs/intro/install/

Context

6 VMs
- 3 Master nodes
- 3 Worker nodes
Resources per node
- 2 VCPUs
- 4 Gio RAM
- 50 Gio Disk
OS
- Debian 11 - 'Bullseye'

Provisionning the cluster:

$ ansible-playbook -i ansible-k3s-cluster/inventory.ini ansible-k3s-cluster/main.yml

Configuring the cluster

Add Flannel CNI
For some reason Flannel is not embedded by default with k3sup
$ kubectl create -f kube/kube-flannel.yaml

If ever the flannel pods crashloopbackoff check the logs, if it's a podCIDR problem check your address ranges and modify the kube-flannel.yml file accordingly in the net-conf.json key

- Get podCIDR
$ kubectl get node k3s-<node> -o=jsonpath='{.spec.podCIDR}'

- Modification of the kube-flannel file, example:
  net-conf.json: |
    {
      "Network": "10.42.4.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

- reload of configuration:
$ kubectl delete -f kube/kube-flannel.yaml 
$ kubectl create -f kube/kube-flannel.yaml

Force restart of CoreDNS
$ kubectl -n kube-system delete pod -l k8s-app=kube-dns
Prevent pod scheduling on masters
$ kubectl taint nodes k3s-master-node-1 k3s-master-node-2 k3s-master-node-3 node-type=master:NoSchedule
Create new NS
$ kubectl create ns monitoring

Install Grafana

Adding the Grafana repository to Helm :
$ helm repo add grafana https://grafana.github.io/helm-charts
Update helm repository:
$ helm repo update
Installing Grafana:
$ helm upgrade --install --namespace monitoring grafana grafana/grafana -f helm-charts/grafana/values.yaml
Get Credentials:
$ kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Create a port-forward to access Grafana locally

$ export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")

$ kubectl --namespace monitoring port-forward $POD_NAME 3000

Access to Grafana:
localhost:3000

Install Victoria Metrics

Adding the Grafana repository to Helm:
$ helm repo add victoria-metrics https://victoriametrics.github.io/helm-charts/

Update helm repository :
$ helm repo update

Installing Victoria Metrics Cluster:
$ helm upgrade --install --namespace monitoring --create-namespace victoria-metrics victoria-metrics/victoria-metrics-cluster -f helm-charts/victoria-metrics/values.yml

Adding the datasource in grafana

In grafana add a new datasource of type Prometheus, rename it Victoria Metrics and give it this URI:
http://victoria-metrics-victoria-metrics-cluster-vmselect.monitoring.svc.cluster.local:8481/select/0/prometheus/

Install VM-Agent

helm upgrade --install --namespace monitoring vm-agent victoria-metrics/vmagent -f helm-charts/vmagent/values.yml

Start operating ! 🚀

SouenMazouin/k3s-cluster-metering-lab