KubeCon EU 2019

This repository contains the demo code for my KubeCon EU 2019 talk about building multi-cloud clusters using WireGuard.

In this demo we will imagine we are a company like Nest that is running object detection processes on video captured by IoT devices. We will run a web-app in the cloud connected to a GPU-powered image detection and labeling service in a different public cloud provider. The web-app will stream video from the IoT device over a WireGuard connection to keep the data safe.

Specifically we will:

create a multi-cloud cluster that spans between DigitalOcean and AWS
create some GPU workers in AWS
run the workload that captures video in a device on the edge, e.g. your host capturing video from the webcam
peer the workload with the cluster in the cloud
run a computer vision process on the video captured by the edge workload
accelerate the computer vision using GPUs in AWS.

Prerequisites

You will need:

DigitalOcean and AWS accounts
Terraform installed
the Kilo commandline utility kgctl installed
WireGuard installed

Getting Started

Modify the provided terraform.tfvars file to suit your project:

$EDITOR terraform.tfvars

Running

Create the infrastructure:

terraform init
terraform apply --auto-approve

Annotate the GPU nodes so Kilo knows they are in their own data center:

for node in $(kubectl get nodes | grep -i ip- | awk '{print $1}'); do kubectl annotate node $node kilo.squat.ai/location="aws"; done

Install the manifests:

kubectl apply -f manifests/

Create the local WireGuard link:

IFACE=wg0
sudo ip link add $IFACE type wireguard
sudo ip a add 10.5.0.1 dev $IFACE
sudo ip link set up dev $IFACE

Generate a key-pair for the WireGuard link:

wg genkey | tee privatekey | wg pubkey > publickey

Create a Kilo Peer on the cluster for the local WireGuard link:

PEER=squat
cat <<EOF | kubectl apply -f -
apiVersion: kilo.squat.ai/v1alpha1
kind: Peer
metadata:
  name: $PEER
spec:
  allowedIPs:
  - 10.5.0.1/32
  publicKey: $(cat publickey)
  persistentKeepalive: 10
EOF

Configure the cluster as a peer of the local WireGuard link:

kgctl showconf peer $PEER > peer.ini
sudo wg setconf $IFACE peer.ini
sudo wg set $IFACE private-key privatekey

Add routes to the cluster's allowed IPs:

for ip in $(kgctl showconf peer $PEER | grep AllowedIPs | cut -f 3- -d ' ' | tr -d ','); do
	sudo ip route add $ip dev $IFACE
done

Run the video capture service on the "edge":

docker run --rm --privileged -p 8080:8080 squat/kubeconeu2019 /mjpeg --bind-addr=:8080

Check out the KubeCon application in a browser!

$BROWSER $(kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' | grep kceu | cut -f 2):8080

Finally, clean everything up:

terraform destroy --auto-approve