So, I've not updated this in a while as I've discovered the joys of Terraform and the glory of programatically creating clusters using Golang/ClusterAPI and other methods. Now, I'm not completely abandoning this but for now there is no guaranteed updates (as you can see it's not been updated in 3+ years). One day I may get around to updating all of this - if I don't I'll archive it in 6-12 months - we'll see. I'm making it public and leaving it here in case anyone wants to play/use/update it/learn form the mistakes it makes. It was something I created back in the day when I was learning Kubernetes and attempting to automate it.
These playbooks are designed to help people who want a Kubernetes cluster and don't want to rely on the cloud providers to get one. The process is still in development and ETCD will soon be separated out from the Master Nodes to allow for better reliability. The idea behind it is to give you a close-to-production including GitOps tools (flux), A Service Mesh (Istio), Monitoring (via Prometheus Operator), Helm and more.
All you need to get going is 1 CentOS 8 machine capable of running VMs OR a collection of bare metal devices running CentOS 7.
If you're going for the VM route, which is the most mature, then you don't really need to do much other than provide a server to use as the VM host.
The playbooks will setup LibVirtD on the builder nodes and deploy the cluster. However for now it's designed to work with one builder, not multiple.
I intend to improve on this soon with a wrapper around virsh to provide a rest API (or find one if it exists) but it depends on time available.
- Cluster creation is managed using kubeadm.
- HA Control plane available with HAProxy provided as the LB.
- LB Provided for workers - I'm looking into creating a "Service Controller" based on the "Cloud Controller Manager" so that LoadBalancer service types can be created.
- Enabled auditing by default.
- Flux - For GitOPs
- StorageOS is the default storage driver. This will put data in /var/lib/storageos/data on each worker node. Storage pools are created on the builder, attach and mount a disk to /var/lib/libvirt/images/storageos to expand on this storage.
- Helm v2 with SSL and RBAC, I'd love to go for v3 but until more charts or moved to v3 and things like Istio and Gitlab K8S integration move to the new version, we're stuck.
- Istio (Considering a switch to linkerd)
Alpha v0.1
- If it breaks your cluster, I'm not responsible - this is a project meant for my cluster and it should work on most clusters but you know, who knows if dragons are real?
- If using bare metal (best for production):
- Make sure you mark the nodes as remote in
host_vars/{HOST}/main.yml. - StorageOS is brand new in this project and going through testing but should work fine.
- You'll be responsible for Installing the OS, I'm not working with any TFTP/iPXe setups yet.
- Make sure you mark the nodes as remote in
- If using LibVirt/KVM:
- The playbooks pretty much do everything.
- You may hit a few initial errors because a python module that I can't install because you'd need THAT python module for me to do it (got that?), will have to be done by you.
- You'll need to install Ansible first. I used
pip install ansibleon my local machine for all of this.
- If you want to use the Kemp Free LB, this will work but the licencing is a bit squiffy so you'll have to sort that yourself.
- I can't pull the image without registration unless I provide some source for it so you'll need to do this, convert it to qcow2 and move it into the LibVirtD templates dir.
- Do a find through the code to uncomment the relevant parts, I'll set this in a VAR this later.
- There seems to be an issue with HAProxy managing persistent connections on the Control Plane at the moment.
Things like
kubectl get po -wwill drop and helm may fail. I'm looking into layer 7 over layer 4 balancing to see if I can negate this.
- Set up Ansible on your machine
- Pull this code and:
- Modify or comment out
hosts.ymlas required. - Configure your hosts within
host_vars/HOST_NAME/main.yml. Check outhost_vars/all/main.yml.examplefor an example. - Make sure you configure your variables in
group_vars/all/main.yml. Check outgroup_vars/all/main.yml.examplefor an example.
- Modify or comment out
- Turn on your Builder machine(s). I haven't gone so far as to build roles for magic packets etc... yet ;-)
- Go!
# --extra-vars available
# build_instances: true|false
## This command uses the root user to initialise the ansible_user on the buidler node.
## All VMs will be auto populated with this user.
$ ansible-playbook adhoc/initialise_ansible_user.yml -k -c paramiko --become --ask-become-pass --ask-pass -c paramiko
## This builds the entire K8S cluster from scratch
$ ansible-playbook site.yml --extra-vars="build_instances=true"There are some ad-hoc playbooks too alongside the adhoc/initialise_ansible_user.yml.
To generate a new token nodes will use for joining an existing cluster - adhoc/generate_new_token.yml
The playbooks can be run per host or as phases. The phases are the default.
This section will install all the required yum/apt packages and python packages that will be required for this project to work.
This will configure the node you define as the builder node in the hosts.yml file. LibVirtd/QEMU etc is configured along with bridge networking.
Deploys the instances used in the cluster via libvirt.
Install required packages on all nodes, configure firewall, hosts file and more common tasks.
Configures the HAProxy load balancers.
Configures NFS storage.
Builds the first controller node for the cluster.
Adds additional controller nodes for the control plane.
Add any worker nodes.
Configure StorageOS on the cluster.
Installs Helm, Istio, Kube-Prometheus (Prometheus Operator) and Flux.