/k3sup

bootstrap Kubernetes with k3s over SSH < 1 min 🚀

Primary LanguageGoMIT LicenseMIT

k3sup 🚀 (said 'ketchup')

k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

The tool is written in Go and is cross-compiled for Linux, Windows, MacOS and even on Raspberry Pi.

How do you say it? Ketchup, as in tomato.

Build Status Go Report Card GoDoc License: MIT GitHub All Releases

What's this for? 💻

This tool uses ssh to install k3s to a remote Linux host. You can also use it to join existing Linux hosts into a k3s cluster as agents. First, k3s is installed using the utility script from Rancher, along with a flag for your host's public IP so that TLS works properly. The kubeconfig file on the server is then fetched and updated so that you can connect from your laptop using kubectl.

You may wonder why a tool like this needs to exist when you can do this sort of thing with bash.

k3sup was developed to automate what can be a very manual and confusing process for many developers, who are already short on time. Once you've provisioned a VM with your favourite tooling, k3sup means you are only 60 seconds away from running kubectl get pods on your own computer. If you are a local computer, you can bypass SSH with k3sup install --local

Uses

  • Bootstrap Kubernetes with k3s onto any VM with k3sup install - either manually, during CI or through cloud-init
  • Get from zero to kubectl with k3s on Raspberry Pi (RPi), VMs, AWS EC2, Packet bare-metal, DigitalOcean, Civo, Scaleway, and others
  • Build a HA, multi-master (server) cluster
  • Fetch the KUBECONFIG from an existing k3s cluster
  • Join nodes into an existing k3s cluster with k3sup join

Bootstrapping Kubernetes

Conceptual architecture Conceptual architecture, showing k3sup running locally against any VM such as AWS EC2 or a VPS such as DigitalOcean.

Download k3sup (tl;dr)

k3sup is distributed as a static Go binary. You can use the installer on MacOS and Linux, or visit the Releases page to download the executable for Windows.

curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/

k3sup --help

k3sup is made available free-of-charge, but you can support its ongoing development through GitHub Sponsors 💪

A note for Windows users

Windows users can use k3sup install and k3sup join with a normal "Windows command prompt".

Demo 📼

In the demo I install Kubernetes (k3s) onto two separate machines and get my kubeconfig downloaded to my laptop each time in around one minute.

  1. Ubuntu 18.04 VM created on DigitalOcean with ssh key copied automatically
  2. Raspberry Pi 4 with my ssh key copied over via ssh-copy-id

Watch the demo:

asciicast

Who is the author? 👏

k3sup is Open Source Software (OSS) and was created by Alex Ellis - the founder of OpenFaaS ® & inlets. Alex is also an active part of the Docker & Kubernetes community as a CNCF Ambassador.

If you've benefitted from his open source projects or blog posts in some way, then and join dozens of other developers today by buying an Insiders Subscription 🏆

  • Buy an Insiders Subscription via GitHub today

Usage ✅

The k3sup tool is a client application which you can run on your own computer. It uses SSH to connect to remote servers and creates a local KUBECONFIG file on your disk. Binaries are provided for MacOS, Windows, and Linux (including ARM).

👑 Setup a Kubernetes server with k3sup

You can setup a server and stop here, or go on to use the join command to add some "agents" aka nodes or workers into the cluster to expand its compute capacity.

Provision a new VM running a compatible operating system such as Ubuntu, Debian, Raspbian, or something else. Make sure that you opt-in to copy your registered SSH keys over to the new VM or host automatically.

Note: You can copy ssh keys to a remote VM with ssh-copy-id user@IP.

Imagine the IP was 192.168.0.1 and the username was ubuntu, then you would run this:

  • Run k3sup:
export IP=192.168.0.1
k3sup install --ip $IP --user ubuntu

Other options for install:

  • --cluster - start this server in clustering mode, for use with dqlite (embedded HA)
  • --skip-install - if you already have k3s installed, you can just run this command to get the kubeconfig
  • --ssh-key - specify a specific path for the SSH key for remote login
  • --local-path - default is ./kubeconfig - set the file where you want to save your cluster's kubeconfig. By default this file will be overwritten.
  • --merge - Merge config into existing file instead of overwriting (e.g. to add config to the default kubectl config, use --local-path ~/.kube/config --merge).
  • --context - default is default - set the name of the kubeconfig context.
  • --ssh-port - default is 22, but you can specify an alternative port i.e. 2222
  • --k3s-extra-args - Optional extra arguments to pass to k3s installer, wrapped in quotes, i.e. --k3s-extra-args '--no-deploy traefik' or --k3s-extra-args '--docker'. For multiple args combine then within single quotes --k3s-extra-args '--no-deploy traefik --docker'.
  • --k3s-version - set the specific version of k3s, i.e. v0.9.1
  • --ipsec - Enforces the optional extra argument for k3s: --flannel-backend option: ipsec

  • See even more install options by running k3sup install --help.

  • Now try the access:

export KUBECONFIG=`pwd`/kubeconfig
kubectl get node

Note that you should always use pwd/ so that a full path is set, and you can change directory if you wish.

Advanced KUBECONFIG options

You can also merge the remote config into your main KUBECONFIG file $HOME/.kube/config, then use kubectl config get-contexts or kubectx to manage it.

The default "context" name for the remote k3s cluster is default, however you can override this as below.

For example:

k3sup install \
  --ip $IP \
  --user $USER \
  --merge \
  --local-path $HOME/.kube/config \
  --context my-k3s

Here we set a context of my-k3s and also merge into our main local KUBECONFIG file, so we could run kubectl config set-context my-k3s or kubectx my-k3s.

😸 Join some agents to your Kubernetes server

Let's say that you have a server, and have already run the following:

export SERVER_IP=192.168.0.100
export USER=root

k3sup install --ip $SERVER_IP --user $USER

Next join one or more agents to the cluster:

export AGENT_IP=192.168.0.101

export SERVER_IP=192.168.0.100
export USER=root

k3sup join --ip $AGENT_IP --server-ip $SERVER_IP --user $USER

That's all, so with the above command you can have a two-node cluster up and running, whether that's using VMs on-premises, using Raspberry Pis, 64-bit ARM or even cloud VMs on EC2.

Create a multi-master (HA) setup

As of k3s 1.0 a HA multi-master configuration is available through dqlite. A quorum of masters will be required, which means having at least three nodes.

  • Initialize the cluster with the first server

Note the --cluster flag

export SERVER_IP=192.168.0.100
export USER=root

k3sup install \
  --ip $SERVER_IP \
  --user $USER \
  --cluster
  • Join each additional server

Note the new --server flag

export USER=root
export SERVER_IP=192.168.0.100
export NEXT_SERVER_IP=192.168.0.101

k3sup join \
  --ip $NEXT_SERVER_IP \
  --user $USER \
  --server-user $USER \
  --server-ip $SERVER_IP \
  --server

Now check kubectl get node:

kubectl get node
NAME              STATUS   ROLES    AGE     VERSION
paprika-gregory   Ready    master   8m27s   v1.16.3-k3s.2
cave-sensor       Ready    master   27m     v1.16.3-k3s.2

👨‍💻 Micro-tutorial for Raspberry Pi (2, 3, or 4) 🥧

In a few moments you will have Kubernetes up and running on your Raspberry Pi 2, 3 or 4. Stand by for the fastest possible install. At the end you will have a KUBECONFIG file on your local computer that you can use to access your cluster remotely.

Conceptual architecture, showing k3sup running locally against bare-metal ARM devices.

  • Download etcher.io for your OS

  • Flash an SD card using Raspbian Lite

  • Enable SSH by creating an empty file named ssh in the boot partition

  • Generate an ssh-key if you don't already have one with ssh-keygen (hit enter to all questions)

  • Find the RPi IP with ping -c raspberrypi.local, then set export SERVER_IP="" with the IP

  • Copy over your ssh key with: ssh-copy-id pi@raspberrypi.local

  • Run k3sup install --ip $SERVER_IP --user pi

  • Point at the config file and get the status of the node:

export KUBECONFIG=`pwd`/kubeconfig
kubectl get node -o wide

You now have kubectl access from your laptop to your Raspberry Pi running k3s.

If you want to join some nodes, run export IP="" for each additional RPi, followed by:

  • k3sup join --ip $IP --server-ip $SERVER_IP --user pi

Remember all these commands are run from your computer, not the RPi.

Now where next? I would recommend my detailed tutorial where I spend time looking at how to flash the SD card, deploy k3s, deploy OpenFaaS (for some useful microservices), and then get incoming HTTP traffic.

Try it now: Will it cluster? K3s on Raspbian

Caveats on security

If you are using public cloud, then make sure you see the notes from the Rancher team on setting up a Firewall or Security Group.

k3s docs: k3s configuration / open ports

If your ssh-key is password-protected

If the ssh-key is encrypted the first step is to try to connect to the ssh-agent. If this works, it will be used to connect to the server. If the ssh-agent is not running, the user will be prompted for the password of the ssh-key.

On most Linux systems and MacOS, ssh-agent is automatically configured and executed at login. No additional actions are required to use it.

To start the ssh-agent manually and add your key run the following commands:

eval `ssh-agent`
ssh-add ~/.ssh/id_rsa

You can now just run k3sup as usual. No special parameters are necessary.

k3sup --ip $IP --user user

Contributing

Insiders Subscription ☕️ 👏

Buy an Insiders Subscription today via GitHub.

Blog posts & tweets

Blogs posts, tutorials, and Tweets about k3sup (#k3sup) are appreciated. Please send a PR to the README.md file to add yours.

Contributing via GitHub

Before contributing code, please see the CONTRIBUTING guide. Note that k3sup uses the same guide as inlets.dev.

Both Issues and PRs have their own templates. Please fill out the whole template.

All commits must be signed-off as part of the Developer Certificate of Origin (DCO)

License

MIT

📢 What are people saying about k3sup?

Checkout the Announcement tweet

Similar tools & glossary

Glossary:

  • Kubernetes: master/slave
  • k3s: server/agent

Related tools:

  • k3s - Kubernetes as installed by k3sup. k3s is a compliant, light-weight, multi-architecture distribution of Kubernetes. It can be used to run Kubernetes locally or remotely for development, or in edge locations.
  • k3d - this tool runs a Docker container on your local laptop with k3s inside
  • kind - kind can run a Kubernetes cluster within a Docker container for local development. k3s is also suitable for this purpose through k3d. KinD is not suitable for running a remote cluster for development.
  • kubeadm - a tool to create fully-loaded, production-ready Kubernetes clusters with or without high-availability (HA). Tends to be heavier-weight and slower than k3s. It is aimed at cloud VMs or bare-metal computers which means it doesn't always work well with low-powered ARM devices.
  • k3v - "virtual kubernetes" - a very early PoC from the author of k3s aiming to slice up a single cluster for multiple tenants
  • k3sup-multipass - a helper to launch single node k3s cluster with one command using a multipass VM and optionally proxy the ingress to localhost for easier development.

Appendix

sudo required for kubectl

It's recommended that you do not run kubectl on your node, or RPi cluster, but use the KUBECONFIG file from your controller/laptop.

See also: Set kubeconfig to world-readable

iptables

Note added by Eduardo Minguez Perez

Currently there is an issue in k3s involving iptables >= 1.8 that can affect the network communication. See the k3s issue and the corresponding kubernetes one for more information and workarounds. The issue has been observed in Debian Buster but it can affect other distributions as well.

Go modules

Troubleshooting

There was an odd edge case in one of the previous versions, If you are having error with helm throwing an error about tiller not being ready then you might want to remove the ~/.k3sup/ directory, which holds some info used by k3sup. Once you have removed this you should try again.

If you are having any other issues or have questions please open an issue.