MinistrBob/k8s_setup

This project contains several playbooks that help you automate setting up a Kubernetes Cluster on VMs or bare-metal servers. `kubeadm` deployment method is used.

k8s_setup

Contents

• k8s_setup project
  • Quick start
  • Supported Linux distribution (distros)
  • Project system requirements
  • Main ideas (basic concept)
  • Playbooks
  • Stuff playbooks (in folder stuff)
  • WARNINGS
  • How to use this project
    • Install Git and Ansible
    • Git clone project
    • Setting up ansible
    • Generate SSH key of any type
    • Configure inventory for the stand
    • Deploy SSH public key to remote hosts (setup SSH passwordless authentication) and visudo user
    • (Optional) Verify the MAC address and product_uuid are unique for every node
    • Running playbook k8s_setup.yml
    • Cluster initialization on one master
    • Copy commands for join masters and workers
    • (Optional) Join masters
    • Join workers
  • Clean up (if something went wrong while creating the cluster)

This project contains several playbooks that help you automate setting up a Kubernetes Cluster on VMs or bare-metal servers. kubeadm deployment method is used Bootstrapping clusters with kubeadm.

Quick start

Follow the steps from point How to use this project in sequence.

Supported Linux distribution (distros)

The playbook supports any Linux distributions, since you can add your own tasklist for each distribution or family of distributions. At the moment, the playbook contains tasklists for CentOS 7, CentOS 8, Debian, Ubuntu. The playbook was previously tested on CentOS 7 but has changed a lot since then. The current version has been tested on Astra Linux 1.7 (similar to Debian 10).
For Astra Linux os_distrib_version = astralinux1.
NOTE: In Astra Linux it is possible to install Kubernetes version no higher than 1.23.4 from the VANILLA repository, because the Astra Linux repository has cri-tools version=1.21.0~2, and kubernetes binaries require cri-tools version >= 1.23.0. In the Astra Linux repository, the maximum Kubernetes version is 1.23.17.

Project system requirements

• Ansible version 2.9+.
• On all servers, a sudo user must be created under which the SSH connection occurs. You yourself can automate the process of creating such a user if it does not exist. When cloning virtual machines from a template, there is usually such an administrator user already there.

Main ideas (basic concept)

• All variables and list of hosts are collected in one file inventory (see example inventory\example.standXXX.yml). In addition to variables dependent on the distribution and OS version. They are in the vars folder in the files vars/{{ os_distrib_version }}.yml
• Main playbook k8s_setup.yml is divided into three stages (each of which is represented by a separate role): OS prepare, Kubernetes setup and HA setup. Each of which can be performed separately (or not performed). This can be regulated by variables in inventory (see example inventory\example.standXXX.yml) and tags.
• Each step in each of the three stages can be performed separately (or not performed). This can be regulated by variables in inventory (see example inventory\example.standXXX.yml) and tags.
• Stage OS prepare helps to prepare the server operating system and can be used not only for servers intended for the Kubernetes, but also for auxiliary servers (group auxiliary) included in this stand, for example, DNS, NTP, etc.
• Stage Kubernetes Setup OS prepare for K8S, setup container runtime and install k8s packages.
• Stage HA Setup installation and configuration of keepalived and haproxy.
• Support for various Linux distributions is implemented by adding a tasklists whose names are given by ansible facts os_distrib_version, os_family_version, os_distrib and os_family (see k8s_setup.yml). Attention! Some tasks for some Linux distributions are not currently implemented, for example Config Access Control system (SELinux, AppArmor, parsec), and are left as a stub (see roles/os-prepare/tasks/config_ac_astralinux.yml).
• Each stage provides support for the execution of PRE and POST tasks. To use this feature, you need to create a file like this in the role folder (possible names are shown): pre_tasks_{{ os_distrib_version }}.yml, pre_tasks_{{ os_family_version }}.yml, pre_tasks_{{ os_distrib }}.yml, pre_tasks.yml. ATTENTION! These files are in .gitignore and are not stored in the git repository.

Playbooks

• k8s_setup.yml - Main playbook for OS prepare, Kubernetes setup and HA setup.
• deploy_ssh_public_key.yml - Set up passwordless access via SSH.
• k8s_init_cluster.yml - K8s cluster initialization (kubeadm init).
• k8s_join_masters.yml - Join masters (for HA).
• k8s_join_workers.yml - Join workers.
• k8s_delete_cluster.yml - Delete k8s cluster (HA is not deleted).

Stuff playbooks (in folder stuff)

• check_unique_uuid.yml - Verify the MAC address and product_uuid are unique for every node. Needed for cloned VMs.
• reboot_servers.yml - Playbook for reboot servers. Example: Reboot all servers except one.

ansible-playbook stuff/reboot_servers.yml -e 'target=all,!pp-bal-01'

!!! WARNINGS !!!

• At stage Prepare OS, the reboot is performed twice, so it is better to execute the playbook from a separate server (administrator's computer), otherwise the playbook execution will be interrupted. See reboot_servers.yml in Stuff playbooks (in folder stuff). If you do not have a separate computer, then you can perform a workaround. Turn off the second and third stages. Set reboot=false. After completing the first stage, first restart the server from which you are running the playbook manually, and then restart all servers using stuff/reboot_servers.yml (see Stuff playbooks (in folder stuff)).
• Step config_os_network assumes that the network in the OS is managed by the networking and resolvconf services. If the network settings are managed by the NetworkManager service, disable this step.
• Virtual IP (VIP) must be recognized via DNS or be included in the /etc/hosts file.

How to use this project

The project requires Ansible, which can be installed both on one of the computers of the current stand and a completely separate computer that has network access to all computers of the stand (admin's computer).

Install Git and Ansible

Installation is different for different Linux distributions, so see the documentation for those software products:

• Installing Git
• Installing Ansible

Attention! For ansible you will need to install additional collections.

ansible-galaxy collection install community.general
ansible-galaxy collection install ansible.posix

Git clone project

cd ~
git clone https://github.com/MinistrBob/k8s_setup.git
cd k8s_setup

Setting up ansible

You can change ansible settings in ansible.cfg file. By default, the settings are optimal.

cp ansible.cfg.example ansible.cfg
nano ansible.cfg

If you have one stand and one inventory then you can define an inventory variable so you don't have to specify it every time.

inventory = inventory/standXXX.yml

Generate SSH key of any type

ssh-keygen -t ed25519

Configure inventory for the stand

All variables in stand.yml.example have comments explaining the purpose of the variables.
The name of the stand (stand.yml) can be anything. It is possible to keep several configuration files here for several stands.
Attention! Two variables are required for SSH options: ansible_user and ansible_private_key_file (see inventory/stand.yml.example).
The project uses three groups:

• kube_masters - Servers for master components Kubernetes. There may be one. For HA cofiguration it is better to have three master servers.
• kube_not_masters - All servers for the cluster Kubernetes (all except the masters).
• auxiliary - Auxiliary stand servers that are not included in the Kubernetes cluster. For them, only the first stage OS prepare is performed, for example, DNS, NTP, etc.

cp inventory/stand.yml.example inventory/standXXX.yml
nano inventory/standXXX.yml

Deploy SSH public key to remote hosts (setup SSH passwordless authentication) and visudo user

(Optional) You can check SSH connection is on one of host manualy to make sure that the ssh connection is established at all.

ssh -i ~/.ssh/id_ed25519 user1@pp-ceph-osd-01

Playbook executed with password.

ansible-playbook -i inventory/standXXX.yml deploy_ssh_public_key.yml --ask-pass

If on the remote server the sudo command asks for a password, then you need to specify it too.
Playbook executed with password and sudo password.

ansible-playbook -i inventory/standXXX.yml deploy_ssh_public_key.yml --ask-pass --ask-become-pass

If even after specifying the correct password you get error Permission denied (publickey,password) then you can try to use module paramiko instead of openssh to connect SSH.
In the future, after the public key is installed, paramiko does not need to be used.

ansible-playbook -i inventory/standXXX.yml deploy_ssh_public_key.yml -c paramiko --ask-pass

(Optional) Verify the MAC address and product_uuid are unique for every node

Playbook check_unique_uuid.yml show MAC addresses and UUID. If VMs were cloned, then they may have not uniqu MAC and UUID. You must visually verify that everything is unique.
NOTE: VMware ESXi changes the MAC address during virtual machine cloning.

ansible-playbook stuff/check_unique_uuid.yml

Running playbook k8s_setup.yml

Note: All kubernetes packages are installed on all servers. They take up little space and don't interfere with anything, so separating the installation of packages into master and non-master doesn't make sense IMHO.
This playbook does all the main work and fully prepares the cluster before it is initialized.

ansible-playbook -i inventory/standXXX.yml k8s_setup.yml

Tags can only be used to execute specific tasks.

ansible-playbook -i inventory/standXXX.yml k8s_setup.yml --tags "prep1,prep2"

Cluster initialization

Examples of three different ways to initialize a cluster. Here extra-vars parameter mc is the cluster initialization command.

ansible-playbook -i inventory/standXXX.yml k8s_init_cluster.yml --extra-vars "mc='kubeadm init'"
ansible-playbook -i inventory/standXXX.yml k8s_init_cluster.yml --extra-vars "mc='kubeadm init --pod-network-cidr=10.244.0.0/16'"
ansible-playbook -i inventory/standXXX.yml k8s_init_cluster.yml --extra-vars "mc='kubeadm init --control-plane-endpoint pp-vip-k8s.mydomen.com:8443 --upload-certs --pod-network-cidr 10.244.0.0/16'"

During the initialization process, two log files will be created:

• cluster_init.log - Cluster initialization log and command for join masters and workers.
• install_calico.log - Pod network installation log.

To check the cluster you can execute (this is as example):

$ kubectl get nodes -o wide
NAME              STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
pp-mk8s-01   Ready    master   39m   v1.19.2   10.147.245.25   <none>        CentOS Linux 7 (Core)   3.10.0-1127.19.1.el7.x86_64   docker://19.3.11

$ kubectl -n kube-system get pod
NAME                                      READY   STATUS    RESTARTS   AGE
calico-kube-controllers-c9784d67d-98dx8   1/1     Running   0          39m
calico-node-ml9gv                         1/1     Running   0          39m
coredns-f9fd979d6-jwdnc                   1/1     Running   0          39m
coredns-f9fd979d6-wgrqh                   1/1     Running   0          39m
etcd-rtz-ppd-mk8s-01                      1/1     Running   0          39m
kube-apiserver-rtz-ppd-mk8s-01            1/1     Running   0          39m
kube-controller-manager-rtz-ppd-mk8s-01   1/1     Running   0          39m
kube-proxy-z4qvt                          1/1     Running   0          39m
kube-scheduler-rtz-ppd-mk8s-01            1/1     Running   0          39m

Copy commands for join masters and workers

Copy command for join masters and workers from screen or from file cluster_init.log to vars/join_commands.yml.

cp vars/join_commands.yml.example vars/join_commands.yml
nano vars/join_commands.yml

(Optional) Join masters

If you are setting high availability (HA) configuration then you need to join other masters to cluster.

ansible-playbook -i inventory/standXXX.yml k8s_join_masters.yml

Join workers

Join all workers servers to cluster.

ansible-playbook -i inventory/standXXX.yml k8s_join_workers.yml

You can check on the master server.

kubectl get nodes

Clean up (if something went wrong while creating the cluster)

See kubernetes documentation: Clean up

ansible-playbook -i inventory/standXXX.yml k8s_delete_cluster.yml

(Optional) If you have a need to clean the iptavles (you can not do this).
WARNING: This can be dangerous! Therefore, you can backup the OS or take a snapshot of the virtual machine beforehand.

sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
sudo iptables -L -n -t nat