Kubernetes Raspberry Pi 4 Cluster

This guide covers the construction, installation and use of a Kubernetes Cluster hosted on a set of Raspberry Pi 4 computers.

Under construction ...

Hardware

A minimum configuration to demonstrate the features of Kubernetes that I used is as follows:

Item	Use	Example	Quantity	Notes
Controller	Raspberry Pi 4 Model B	Amazon Link	1	Any RPi with 4 cores and 1GB RAM+
Workers	Raspberry Pi 4 Model B	Amazon Link	3	Use a model with 4GB RAM and Quad Core CPU
Power	Anker PowerPort 60 W 6-Port	Amazon Link	1
USB Cables	Power to Pi's	Amazon Link	1 Pack	Need 4
USB C Adaptors	Power to Pi's	Amazon Link	1 Packs	Need 4
Switch	Network / 8 ports	Amazon Link	1	Only 5 ports needed. Note 1
Ethernet Cables	Network	Amazon Link	1 Pack	Need 4
USB to 12 Converter	Switch Power	Amazon Link	1	Note 2
Frame	Somewhere for your Pi's to live	Amazon Link	2
Storage	O/S Storage	Amazon Link	4	Cheap and seem to work ok

Note 1 Netgear also produce a 5 port switch that runs from 5V. (8 port needs 12V). I wanted to run a bigger cluster so went for 8 ports.

Note 2 The 5V to 12V cable is just to make things tidier. You can use the original 12V PSU for the switch. Be careful to check power requirements if not using the original PSU.

Configure Raspberry Pi Nodes

Software versions:

O/S version: Raspbian Buster Lite / Kernel 4:19
Kubernetes version: v1.15.0
Docker version: 18.09.0

Network:

Master node: 172.16.1.210 (Host name: kubectl4)
Worker node: 172.16.1.211..214 (Host names: worker41..44)
Ingress node: 172.16.1.211
Gateway: 172.16.1.1
DNS: 1.1.1.1, 8.8.8.8

Install O/S

Install Raspbian Buster Lite Download onto an SD card and boot
Update networking to set a static IP, Gateway and DNS addresses (IPV4 only for now) by editing this file

sudo nano /etc/dhcpcd.conf

Disable the swap file

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo systemctl disable dphys-swapfile

Run the config tool (sudo raspi-config), set a new password, enable sshand set the hostname
Reboot (Must do this now due to host name change)
Install Docker

sudo curl -sL get.docker.com |  sh
sudo usermod -aG docker pi

Add Kubernetes repository

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - 
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Note No specific Buster release is available at the time of writing. However, the Xenial version appears to work without issue.

Edit boot configuration

sudo nano /boot/cmdline.txt

...and add the following options

ipv6.disable=1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Update software

sudo apt update && sudo apt upgrade -y && sudo apt autoremove -y

Configure network bridge (Not sure this is necessary on later kernels ....)

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Check here for order: https://github.com/raynix/ansible-kubeadm/blob/master/roles/kubeadm/tasks/main.yaml

Reboot

This gives a basic configuration ready to install Kubernetes on

Install Kubernetes

sudo apt-get install -y kubeadm

This is now a complete base image which can be used for control and worker nodes. To save time, make an image copy to all of your SD cards. Don't forget to change IP addresses and host names !

Sort out ip tables

A problem exists with Raspbian Buster and iptables, which is now based on nftables. Unfortunately Kubernetes doesn't get on with this change so we have to revert to iptables.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy 
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy 
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy

Install Kubernetes Master Node

Select the machine you are going to use as the master node

Load images

kubeadm config images pull

Install (flannel)

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Once complete, note the line starting

kubeadm join --token

You will need this later to join worker nodes to the cluster (Note 1)

Make the install generally available

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Note 1: You can always regenerate the token at a later date by using:

kubeadm token generate
kubeadm token create <token printed from previous command> --print-join-command --ttl=0

Configure Networking

Most guides use Flannel as the networking layer.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Show system to confirm install

kubectl get pods --all-namespaces
kubectl get nodes

Install Kubernetes Worker Node

To add a worker node to the cluster, execute a kubeadm join command using the values generated earlier.

If the master node was at 172.16.1.210, the command would be in the form:

sudo kubeadm join 172.16.1.210:6443 --token <token here>  --discovery-token-ca-cert-hash sha256:<hash value here>

Add the other worker nodes
On the master node, execute kubectl get nodes to show the complete cluster

The cluster is now complete

Publish a Basic Test Service

As a demonstration, we will now deploy the Hypriot Busybox service (A Raspberry Pi compatible Docker Image with a minimal Busybox httpd web server )

Deploy the Service

Deploy service on 2 containers

kubectl run hypriot --image=hypriot/rpi-busybox-httpd --replicas=2 --port=80
kubectl expose deployment hypriot --port 80
kubectl get endpoints hypriot

Verify Deployment on Worker Nodes

curl <IP address of an endpoint (Not a worker node IP address)>

<html>
<head><title>Pi armed with Docker by Hypriot</title>
  <body style="width: 100%; background-color: black;">
    <div id="main" style="margin: 100px auto 0 auto; width: 800px;">
      <img src="pi_armed_with_docker.jpg" alt="pi armed with docker" style="width: 800px">
    </div>
  </body>
</html>

* Here be Dragons ! *

* Everything Past Here is Untested *

Deploy a Loader Balancer and Ingress Point

Now we have the service running we need to configure a single ingress point with a load balancer.

Install load balancer

kubectl apply -f https://raw.githubusercontent.com/hypriot/rpi-traefik/master/traefik-k8s-example.yaml
kubectl label node <worker node to host load balancer> nginx-controller=traefik

Create an ingress point

cat > hypriot-ingress.yaml <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hypriot
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: hypriot
          servicePort: 80
EOF

kubectl apply -f hypriot-ingress.yaml

Verify all nodes are connected and running

Verify correct operation by pointing a browser at the ingress node:

...and this probably won't work. Recent changes in Kubernetes have introduced Role Based Access Control (RBAC)

How to Bypass RBAC

For running test configurations, using RBAC can be painful. A (not recommended for production) way around this is:

kubectl create clusterrolebinding varMyClusterRoleBinding --clusterrole=cluster-admin --serviceaccount=kube-system:default

A reboot of the cluster will probably be needed after this.

Note: From a cold start, the cluster takes approximnately 5 minutes before the ingress responds with the Hypriot web page

codesqueak/k18srpi4