Imagine... Your own testbed for Kubernetes with cheap Raspberry Pis and friends.
Then, lets go!
The first thing you will do, is to create a SD Card for your Pi. Alternatively, you may use the .deb
deployment
Supported OSes/boards:
- Arch Linux ARM (archlinux)
- Raspberry Pi 1 A, A+, B, B+, (ZERO,) armv6 (rpi)
- Raspberry Pi 2 Model B, armv7 (rpi-2)
- Parallella armv7, read more (parallella)
- Cubietruck, armv7 (cubietruck)
- Banana Pro, armv7 (bananapro)
- Hypriot OS (hypriotos)
- Raspberry Pi 1 A, A+, B, B+, armv6 (rpi)
- Raspberry Pi 2 Model B, armv7 (rpi-2)
Windows downloads
Prebuilt SD Card images for Windows:
- archlinux/rpi/v0.6.0
- archlinux/rpi-2/v0.6.0
# Go to our home folder, if you want
cd ~
# Install git if needed and download this project
# sudo apt-get install git
git clone https://github.com/luxas/kubernetes-on-arm
# Change to that directory
cd kubernetes-on-arm
# See which letter your newly inserted SD Card has:
sudo fdisk -l
# Get some help text about supported options
sdcard/write.sh
# Template:
sudo sdcard/write.sh /dev/sdX [board] [os] [rootfs]
# Example: Write the SD Card for Raspberry Pi 2, Arch Linux ARM and include this project's Kubernetes scripts
sudo sdcard/write.sh /dev/sdX rpi-2 archlinux kube-systemd
# The installer will ask you if you want to erase all data on your card
# Answer y/n on that question
# Prepend the command with QUIET=1 if no security check should be made
# Requires an internet connection
# This script runs in 3-4 mins
Boot your board and log into it.
Arch Linux users:
- The user/password is: root/root or alarm/alarm
- These scripts requires root. So if you login via alarm, then
su root
when you´re going to do some serious hacking :)
HypriotOS users:
- The user/password is: pi/raspberry or root/hypriot
- Remember to prepend all commands with
sudo
if you are thepi
user
Yes, I know. Root enabled via ssh isn´t that good. But the task to enhance ssh security is left as an exercise to the user.
# This script will install and setup docker etc.
kube-config install
# First, it will update the system and install docker
# Then it will download prebuilt Kubernetes binaries
# Later, if you build kubernetesonarm/build, then all binaries will be replaced with that version
# The script will ask you for timezone. Defaults to Europe/Helsinki
# Run "timedatectl list-timezones" before to check for values
# It will ask you if it should create a 1 GB swapfile.
# If you are gonna build Kubernetes on your own machine, you have to create this
# It will ask for which hostname you want. Defaults to kubepi.
# Last question is whether you want to reboot
# You must do this now, otherwise docker will behave very strange and fail
# I've built docker v1.8.2 statically, if you want to use that binary instead of pacman's prepend the command with STATICALLY_DOCKER=1 like the other variables. This is experimental.
# If you want to run this script non-interactively, do this:
# TIMEZONE=Europe/Helsinki SWAP=1 NEW_HOSTNAME=mynewpi REBOOT=0 kube-config install
# This script runs in 2-3 mins
If you want to change something in the source, edit files in /etc/kubernetes/source/images
and run kube-config build-images
before you do this
These script are important in the setup process.
They spin up all required services in the right order.
If you skipped the build process, this may take ~10min, depending on your internet connection.
# To enable the master service, run
kube-config enable-master
# To enable the worker service, run
kube-config enable-worker [master-ip]
If you have already made a SD Card and your device is up and running, what can you do instead?
For that, I've made a .deb
package, so you could install it easily
If you already have set up your Pi with latest Raspbian OS for example, follow this guide.
# The OS have to be systemd based, e. g. HypriotOS, Debian Jessie, Arch Linux ARM, Ubuntu 15.04
# Download the latest package
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/kube-systemd.deb > kube-systemd.deb
# or
wget https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/kube-systemd.deb
# Requires dpkg, which is preinstalled in at least all Debian/Ubuntu OSes
dpkg -i kube-systemd.deb
# Setup the enviroinment
# It will ask which board it's running on and which OS
# If your OS is Hypriot or Arch Linux, choose that. Otherwise, choose systemd
# It will download prebuilt binaries
# And make a swap file if plan to compile things
# A reboot is required for it to function properly
kube-config install
## ----- REBOOT -----
# Start the master or worker
kube-config enable-master
kube-config enable-worker [master ip]
# Get some info about the node
kube-config info
With this script, the required docker images are built, then the Kubernetes binaries and last, the Kubernetes images used when running.
# Build all required images
kube-config build-images
# Build all addons
kube-config build-addons
# These scripts will run approximately 45 min on a Raspberry Pi 2
# Grab yourself a coffee during the time!
The script will produce these Docker images:
- luxas/raspbian: Is a stripped
resin/rpi-raspbian
image. - luxas/alpine: Is a Alpine Linux image. Only 8 MB. Based on
mini-containers/base
source. - luxas/go: Is a Golang image, which is used for building repositories on ARM.
- kubernetesonarm/build: This image downloads all source code and builds it for ARM.
These core images are used in the cluster:
- kubernetesonarm/etcd:
etcd
is the data store for Kubernetes. Used only on master. Docs - kubernetesonarm/flannel:
flannel
creates the Kubernetes overlay network. Docs - kubernetesonarm/hyperkube: This is the core Kubernetes image. This one powers your Kubernetes cluster. Docs
- kubernetesonarm/pause:
pause
is a image Kubernetes uses internally. Docs
# See which commands kubectl and kube-config has
kubectl
kube-config
# Get info about your machine and Kubernetes version
kube-config info
# Make an replication controller with an image
# Hopefully you will have some minions, so you is able to see how they spread across hosts
# The nginx-test image will be downloaded from Docker Hub and is a nginx server which only is serving the message: "<p>WELCOME TO NGINX</p>"
kubectl run my-nginx --image=luxas/nginx-test --replicas=3
# The pull might take some minutes
# See that the nginx container is running
docker ps
# See which pods are running
kubectl get pods
# See which nodes we have
kubectl get nodes
# Expose the replication controller "my-nginx" as a service
kubectl expose rc/my-nginx --port=80
# If you want to expose your application on the host, use the external ip argument
kubectl expose rc/my-nginx --port=80 --external-ip=$(hostname -i)
# See which ip we may ping, by getting services
kubectl get svc
# See if the nginx container is working
# Replace $SERVICE_IP with the ip "kubectl get svc" returned
curl $SERVICE_IP
# --> <p>WELCOME TO NGINX</p>
# Start dns, this will spin up 4 containers and expose them as a DNS service at ip 10.0.0.10
# 10.0.0.10 is already enabled as a DNS server in your system, see the file /etc/systemd/resolved.conf.d/dns.conf
# That file makes /etc/resolv.conf use kube-dns also outside of your containers
kube-config enable-addon dns
# See which internal cluster services that are running
kubectl --namespace=kube-system get pods,rc,svc
# Test dns
curl my-nginx.default.svc.cluster.local
# --> <p>WELCOME TO NGINX</p>
# By default, "search [domains]" is added to /etc/resolv.conf
# In this case, these domains are searched: "default.svc.cluster.local svc.cluster.local cluster.local"
# That means, that you may only write "my-nginx", and it will search in those domains
curl my-nginx
# --> <p>WELCOME TO NGINX</p>
# Start the registry
kube-config enable-addon registry
# Wait a minute for it to start
kubectl --namespace=kube-system get pods
# Tag an image
docker tag my-name/my-image registry.kube-system:5000/my-name/my-image
# And push it to the registry
docker push registry.kube-system:5000/my-name/my-image
# On another node, pull it
docker pull registry.kube-system:5000/my-name/my-image
# The registry address may be written longer if search isn't specified.
# registry.kube-system.svc.cluster.local -> registry.kube-system
# The master also proxies the services so that they are accessible from outside
# The -L flag is there because curl has to follow redirects
# You may also type this URL in a web browser
curl -L http://[master-ip]:8080/api/v1/proxy/namespaces/default/services/my-nginx
# Generic apiserver proxy URL
# curl -L http://[master-ip]:8080/api/v1/proxy/namespaces/[namespace]/services/[service-name]
# Check for open ports
netstat -nlp
# See cluster info
kubectl cluster-info
# See master health in a web browser
# cAdvisor in kubelet provides a web site that outputs all kind of stats in real time
# http://$MASTER_IP:4194
# Disable this node. This always reverts the "kube-config enable-*" commands
kube-config disable-node
# Remove the data for the cluster
kube-config delete-data
If you already have set up a lot of devices and already are familiar with one OS, just grab the binaries here, pull the images from Docker Hub and start to hack your own solution 😄
# Get the binaries and put them in /usr/bin
curl -sSL https://github.com/luxas/kubernetes-on-arm/releases/download/v0.6.2/binaries.tar.gz | tar -xz -C /usr/bin
# Pull the images for master
docker pull kubernetesonarm/hyperkube
docker pull kubernetesonarm/etcd
docker pull kubernetesonarm/flannel
docker pull kubernetesonarm/pause
# Pull the images for worker
docker pull kubernetesonarm/hyperkube
docker pull kubernetesonarm/flannel
docker pull kubernetesonarm/pause
Then check the service files here for the right commands to use: https://github.com/luxas/kubernetes-on-arm/tree/v0.6.2/sdcard/rootfs/kube-systemd/usr/lib/systemd/system
However, only use this method if you know what you are doing and want to customize just for your need
Two addons is available right now
- Kubernetes DNS:
- Every service gets the hostname:
{{my-svc}}.{{my-namespace}}.svc.cluster.local
- The default namespace is
default
(surprise), so unless you manually edit something it will land there - Kubernetes internal addon services runs in the namespace
kube-system
- Example:
my-awesome-webserver.default.svc.cluster.local
or justmy-awesome-webserver
may resolve to ip10.0.0.154
- Those DNS names is available both in containers and on the node itself (kube-config automatically adds the info to
/etc/resolv.conf
) - If you want to access the Kubernetes API easily,
curl -k https://kubernetes
orcurl -k https://10.0.0.1
if you remember numbers better (-k
stands for insecure as apiserver has no signed certs by default) - The DNS server itself has allocated ip
10.0.0.10
- The DNS domain is
cluster.local
- Every service gets the hostname:
- Central image registry
- A registry for storing cluster images if e.g. the cluster has no internet connection for a while
- Or for cluster-specific images that one not want to publish on Docker Hub
- This service is available at this address:
registry.kube-system
when DNS is enabled - Just tag your image:
docker tag my-name/my-image registry.kube-system:5000/my-name/my-image
- And push it to the registry:
docker push registry.kube-system:5000/my-name/my-image
kube-ui
was removed, because the Kubernetes team shifted focus to dashboard.
apiserver
proxy:- This is enabled by default by apiserver
- Type this URL in a browser or use
curl
curl -L http://[master-ip]:8080/api/v1/proxy/namespaces/[namespace]/services/[service-name]
- You may build a proxy in front of this with
nginx
that forwards all requests to the apiserver proxy
- Connect a computer to the
flannel
network- It's possible to start
flannel
andkube-proxy
on another computer in the same network and access all services - Run these two commands from a
amd64
machine with docker:docker run --net=host -d --privileged -v /dev/net:/dev/net quay.io/coreos/flannel:0.5.5 /opt/bin/flanneld --etcd-endpoints=http://$MASTER_IP:4001
docker run --net=host -d --privileged gcr.io/google_containers/hyperkube:v1.1.2 /hyperkube proxy --master=http://$MASTER_IP:8080 --v=2
'
- And replace $MASTER_IP with the actual ip of your master pi
- The consuming
amd64
computer can access all services - For example:
curl -k https://10.0.0.1
- It's possible to start
- Make a
service
withexternalIP
- Via
kubectl
:kubectl expose rc {some rc} --port={the port this service will listen on} --container-port={the port the container exposes} --external-ip={the host you want to listen on}
- Example:
kubectl expose rc my-nginx --port=9060 --container-port=80 --external-ip=192.168.200.100
- This will make the service accessible at
192.168.200.100:9060
- Via
Go to a web browser and type: {IP of you Pi node}:4194
and a nice dashboard will be there and show you some nice real-time stats.
The kube-systemd
rootfs uses systemd services for starting/stopping containers.
Systemd services:
- system-docker: Or
docker-bootstrap
. Used for runningetcd
andflannel
. - etcd: Starts the
kubernetesonarm/etcd
container. Depends onsystem-docker
. - flannel: Starts the
kubernetesonarm/flannel
container. Depends onetcd
. - docker: Plain docker service. Dropins are symlinked. Depends on
flannel
. - k8s-master: Service that starts up the main master components
- k8s-worker: Service that starts up
kubelet
and theproxy
.
Useful commands for troubleshooting:
systemctl status (service)
: Get the status for a servicesystemctl start (service)
: Start a servicesystemctl stop (service)
: Stop a servicesystemctl cat (service)
: See the.service
files for an unit.journalctl -xe
: Get the system logjournalctl -xeu (service)
: Get logs for a service
There is a configuration file: /etc/kubernetes/k8s.conf
, but it has only two options:
K8S_MASTER_IP
: Points to the master in the cluster. If the node is master, it uses127.0.0.1
(akalocalhost
). Default:127.0.0.1
FLANNEL_SUBNET
: The subnetflannel
should use. More information. Default:10.1.0.0/16
You can also customize the master containers´ flags in the file: /etc/kubernetes/static/master/master.json
. There the configuration for the master components are. Official file
You can put more .json
files in /etc/kubernetes/static/master
and /etc/kubernetes/static/worker
if you want; they will come up as static pods.
On Arch Linux, this file will override the default eth0
settings. If you have a special eth0
setup (or use some other network), edit this file to fit your use case: /etc/systemd/network/dns.network
If your cluster won't start, try kube-config delete-data
. That will remove all data you store in /var/lib/kubelet
and /var/lib/kubernetes
. If you don't want to delete all data, but have to get Kubernetes up and running, you can answer M
, when running kube-config delete-data
and it will rename the affected directories like this: {,old}
.
This project is under development.
I develop things on the dev
branch
See the ROADMAP
This project ports Kubernetes to the ARM architecture. The primary boards used for testing is Raspberry Pi 2´s.
My goal for this project is that it should be as small as possible, while retaining its flexibility.
It should also be as easy as possible for people, who don´t know anything about Kubernetes, to get started.
I also have opened a proposal for Kubernetes on ARM: kubernetes/kubernetes#17981.
The long-term goal most of this functionality should be present in core Kubernetes.
It should be easy in the future to add support for new boards and operating systems.