Configure a 4 node k8s with Vagrant and libvirt VM provider
This guide helps to setup a 4 node k8s cluster using k3s and also provides guidelines to attach and detach virtual block devices
Pre-requisites
This guide expects the following components to be installed
KVM and its utilities :-
sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
Verify the installation using
virsh list --all
Start libvirtd service
sudo systemctl enable --now libvirtd
Verify by checking the status
sudo systemctl status libvirtd
Install virt-manager
sudo apt install virt-manager
Verify by starting the GUI
sudo virt-manager
Install vagrant
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vagrant
Spin up VMs using vagrant
Clone the repository
git clone
cd vagrant
Spin up VMs using vagrant
vagrant up
Verify the VMs by
vagrant global-status
the master and the 4 nodes should be in running
state
For eg,
id name provider state directory
-------------------------------------------------------------------------------
17d3cdf node1 libvirt running /home/praveen/go/src/github.com/minio/vagrant
3f24371 node3 libvirt running /home/praveen/go/src/github.com/minio/vagrant
7bf13d4 node4 libvirt running /home/praveen/go/src/github.com/minio/vagrant
2ba66d5 node2 libvirt running /home/praveen/go/src/github.com/minio/vagrant
204f6b1 master libvirt running /home/praveen/go/src/github.com/minio/vagrant
Setup k8s on the master and nodes
- Setup k3s on master
SSH into the master node
vagrant ssh master
Setup k3s
curl -sfL https://get.k3s.io | sh -
Save the kubeconfig
cat /etc/rancher/k3s/k3s.yaml
Copy the k3s token from the master node (to be used later)
cat /var/lib/rancher/k3s/server/node-token
Make a note of the master's ip
ip a s
Setup k3s on worker nodes (to be executed on each worker node)
curl -sfL https://get.k3s.io | K3S_URL=https://<ip_of_the_master_node>:6443 K3S_TOKEN=<copied_token_from_master> sh -
/etc/hosts/
Add the following host entries in 100.0.0.1 master.k8s.com master
100.0.0.2 node1.k8s.com node1
100.0.0.3 node2.k8s.com node2
100.0.0.4 node3.k8s.com node3
100.0.0.5 node4.k8s.com node4
(the master can also update the host entries accordingly)
Check the cluster's health
Configure the kubectl in your client/host machine pointing to the cluster
Eg,
export KUBECONFIG=k3s.yaml
Check the status of the nodes by
kubectl get nodes
Add LVs to the nodes
The following script will create one LV of size 800MiB. Feel free to customize as per your needs.
sudo truncate --size=1G /tmp/disk-1.img
sudo losetup --find /tmp/disk-1.img
device=($(sudo losetup --noheadings --output NAME --associated /tmp/disk-1.img))
sudo pvcreate "${device}"
sudo vgcreate "vg-0" "${device}"
sudo lvcreate --name=lv-0 --size=800MiB vg-0
To remove the LVs,
sudo lvremove vg0
sudo vgremove vg0
sudo pvremove <loop1> <loopN>..
sudo losetup --detach-all
virsh
Add virtual block devices using The following steps will attach a block device of size 1G to the respective node
STEP 1: Create a raw image using qemu-img
qemu-img create -f raw "vdb.img" 1G
STEP 2: Create a config file (.xml)
<disk type='file' device='disk'>
<source file='vdb.img'/>
<target dev='vdb'/>
<serial>node1vdbserial</serial>
</disk>
(Note: Save the file in .xml)
STEP 3: Attach the device using virsh
in node1
sudo virsh attach-device vagrant_node1 <file>.xml --live --persistent
STEP 4: SSH into the node
lsblk
Check if a new block device has been added
To detach the device,
sudo virsh detach-device vagrant_node1 <file>.xml