Declaratively build a 4 node Kubernetes cluster on Proxmox using Ansible and QEMU. Optionally enable advanced features including ingress, load balancing, etc.
Approximate deployment time: 25 minutes
- Proxmox server
- DNS Server
- Ansible 2.7.0+. Known incompatibility with a previous build.
Required:
- Modify the
vars.yml
file with values specific to your environment. - Provision DNS A records for the IP Addresses & Hostnames you defined for your nodes in the
vars.yml
file. - Modify the
inventory.ini
file to reflect your chosen DNS records and the location of the SSH keys used to connect to the nodes. - Run the deployment:
ansible-playbook -i inventory.ini site.yml
- After deployment, a
~/.kube
directory will be created on your workstation. Within yourconfig
anauthentication_token
file can be be found. This token is used to authenticate against the Kubernetes API and Dashboard using your account. To connect to the dashboard, installkubectl
on your workstation and runkubectl proxy
then navigate to the Dashboard Endpoint in your browser.
Optional:
To enable an optional feature, fill in the additional parameters in vars.yml
and execute a playbook listed below.
Feature | Command | Requirements |
---|---|---|
NFS backed persistent storage | ansible-playbook -i inventory.ini playbooks/optional/deploy_nfs_provisioner.yml |
|
MetalLB Load Balancer | ansible-playbook -i inventory.ini playbooks/optional/deploy_metallb.yml |
|
NGINX Ingress Controller | ansible-playbook -i inventory.ini playbooks/optional/deploy_ingress-nginx.yml |
MetalLB or other Load Balancer integration |
DataDog agents | ansible-playbook -i inventory.ini playbooks/optional/deploy_datadog.yml |
- You can rollback the entire deployment with:
ansible-playbook -i inventory.ini playbooks/optional/delete_all_resources.yml
- If Calico isn't deploying correctly it's likely the CIDR you assigned to it in
vars.yml
conflicts with your network. - See this repository to do this with LXC instead. Benefits of using LXC include:
* No virtualization overhead means better performance
* Ability to directly mount volumes from your server into your containers.
- Add better support for multi-node Proxmox clusters.
- Perform security audit and enhance if necessary.
- Add info to README about updating inventory file and how to handle SSH key generation and propegation.
- Add playbook to integrate k8s with a log server.
- Automatically include
inventory.ini
somehow so one can simply runansible-playbook site.yml
to deploy. - Create playbook to upgrade kubernetes version for kubeadm cluster.
- Create playbook to install OS updates on nodes.
- Move dashboard deployment to optional features.
- Refactor lazy sleep tasks for more intelligent
wait_for
tasks.
- The
proxmox_kvm
module is out of date and does not support cloudinit related api calls. Meaning shell commands must be used instead to performqm create
tasks. - The
k8s
module does not support applying Kubernetes Deployments from URL. Instead of usingget_url
to download them first, and then apply them withk8s
, I just useshell
to run akubectl apply -f
. Feature Request here. - Miscellaneous
qcow2
image issues:
OS | Issue |
---|---|
Debian | Kernel Panic on the first boot. Bypassed by stopping and starting a VM after 30 seconds. |
CentOS | A nameserver is baked into /etc/resolv.conf by default. Bug Report here |
CoreOS | Proxmix issued cloud-init does not seem to configure networking properly. |
Ubuntu | Kernel Panic on the first boot. Bypass hack is untested as I prefer Debian. |