k8s-ipsec-cni

Sample implementation of a 3-node Kubernetes cluster using strongSwan and a host-local CNI plugin to provide IPSEC-based networking for pods.

The CNI plugin configuration is mostly taken from the CNI plugin reference.

Requirements:

Working installation of Hashicorp Vagrant 2 (https://www.vagrantup.com/) with either of these provisioners:
- virtualbox
- libvirt

Deployment

Check out this repo into a directory: git clone https://github.com/webvictim/k8s-ipsec-cni
cd into the directory: cd k8s-ipsec-cni
Run vagrant up --provision

The initial startup/provisioning process takes around 10-15 minutes depending on the speed of the test machine and the internet connection.

Once the provisioning process is complete, SSH into the Kubernetes master with vagrant ssh kube-node1.

We should then be able to see three working nodes:

vagrant@kube-node1:~$ kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
kube-node1   Ready     master    14m       v1.10.0
kube-node2   Ready     <none>    11m       v1.10.0
kube-node3   Ready     <none>    9m        v1.10.0

To test, we'll run 9 replicas of the kubernetes-bootcamp image as found here - this will distribute 3 pods per node:

kubectl run kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1 --port=8080 --replicas 9

Then view the status of the deployment:

vagrant@kube-master:~$ kubectl get deployments
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kubernetes-bootcamp   9         9         9            9           19m

Getting info about the pods will show us the IP address assigned to each pod - these are taken sequentially from the CNI range on each node by the plugin:

vagrant@kube-node1:~$ kubectl get pods -o wide
NAME                                   READY     STATUS    RESTARTS   AGE       IP           NODE
kubernetes-bootcamp-5c69669756-2kpdw   1/1       Running   0          25s       10.128.2.6   kube-node2
kubernetes-bootcamp-5c69669756-c497b   1/1       Running   0          25s       10.128.3.9   kube-node3
kubernetes-bootcamp-5c69669756-d4fjf   1/1       Running   0          25s       10.128.3.8   kube-node3
kubernetes-bootcamp-5c69669756-dcwhv   1/1       Running   0          25s       10.128.2.7   kube-node2
kubernetes-bootcamp-5c69669756-mb4qt   1/1       Running   0          25s       10.128.1.5   kube-node1
kubernetes-bootcamp-5c69669756-mq7hp   1/1       Running   0          25s       10.128.3.7   kube-node3
kubernetes-bootcamp-5c69669756-pxsmg   1/1       Running   0          25s       10.128.1.3   kube-node1
kubernetes-bootcamp-5c69669756-v6plw   1/1       Running   0          25s       10.128.2.8   kube-node2
kubernetes-bootcamp-5c69669756-x8sc4   1/1       Running   0          25s       10.128.1.4   kube-node1

If we SSH to any of the nodes, we can ping the IP of any pod on any node:

$ vagrant ssh kube-node1
vagrant@kube-node1:~$ ping 10.128.3.9
PING 10.128.3.9 (10.128.3.9) 56(84) bytes of data.
64 bytes from 10.128.3.9: icmp_seq=1 ttl=63 time=0.950 ms

The kubernetes-bootcamp container listens on port 8080 - we can make HTTP requests to any pod from any node too:

vagrant@kube-node2:~$ curl http://10.128.1.4:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5c69669756-x8sc4 | v=1

vagrant@kube-node3:~$ curl http://10.128.2.6:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5c69669756-2kpdw | v=1

We can also set up a nodePort with kubectl expose:

vagrant@kube-node1:~$ kubectl expose deployment kubernetes-bootcamp --type=NodePort --port 8080
service "kubernetes-bootcamp" exposed
vagrant@kube-node1:~$ kubectl describe services kubernetes-bootcamp
Name:                     kubernetes-bootcamp
Namespace:                default
Labels:                   run=kubernetes-bootcamp
Annotations:              <none>
Selector:                 run=kubernetes-bootcamp
Type:                     NodePort
IP:                       10.105.161.103
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  31990/TCP
Endpoints:                10.128.1.3:8080,10.128.1.4:8080,10.128.1.5:8080 + 6 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

When we make a curl request to the nodePort, we see the requests getting routed around between the different pods:

vagrant@kube-node1:~$ curl http://10.128.1.1:31990
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5c69669756-dcwhv | v=1
vagrant@kube-node1:~$ curl http://10.128.1.1:31990
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5c69669756-c497b | v=1
vagrant@kube-node1:~$ curl http://10.128.1.1:31990
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5c69669756-x8sc4 | v=1
vagrant@kube-node1:~$ curl http://10.128.1.1:31990
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-5c69669756-mb4qt | v=1

The nodes can be freely rebooted and their IPSEC connections will be re-established when the node comes back up, giving us a full mesh setup. The same is true if we run ipsec restart, ipsec down mesh-left or similar - strongSwan is set up to always re-establish the tunnels.

Notes and observations

IP exhaustion
- Using the IPAM host-local plugin with the standard data directory (/var/lib/cni) means that eventually the pool of host-local IPs will be depleted as it's never cleaned up. We solve this by overriding the 'dataDir' properly in the host-local section and using a directory under /run instead (which will be cleared whenever the node is rebooted)
Kubelet/strongSwan start order
- When Kubelet starts, it doesn't bring up the cni0 bridge until a pod is started that needs to use CNI. This means that strongSwan won't work until it's restarted after the first pod is created. This is quite messy, so we get around this by creating the cni0 bridge ourselves and assigning the correct IP address to it before Kubelet or strongSwan are started. We also add a systemd drop-in override file to specify that strongSwan should always be started before kubelet.service to make sure that the VPN tunnels are up before Kubelet needs to communicate with the master.
iptables/traffic forwarding issues
- By default, the policy on the FORWARD chain is DROP which stops us from communicating from pod to pod across the IPSEC tunnel.
- We add a startup script to bring the cni0 bridge back up correctly on restart and also to re-add the firewall rules to the FORWARD chain which allow traffic to flow in/out of cni0 over the IPSEC tunnels. We could probably accomplish something similar with iptables-persistent but Kubernetes and Docker both like to have control of iptables rules so it seems easier just to re-add the rules ourselves and let them keep control of the bigger picture.
- strongSwan appears to have some issues when it's set up to use policy based routing where traffic actually flows in the clear rather than going over the IPSEC tunnels. To combat this, we stopped using policy-based IPSEC and switched to route-based IPSEC instead where we create a tunnel and explicitly tag traffic to be sent over it.

webvictim/k8s-ipsec-cni

k8s-ipsec-cni

Deployment

Notes and observations