-
Create 2 or more VPS/VM with following minimum requirement:
Master Worker1 Core 4 2 Ram 4GB 4GB Storage 20GB+10gb extend 20GB + 10GB extend Operation Ubuntu 20.04+ Ubuntu 20.04+ Network IP 172.16.171.135 (example) 172.16.171.133 (example)
-
Install some tools in Ubuntu:
sudo apt update sudo apt install vim git curl -y
-
Clone setup files and setup K8s for master node:
git clone https://github.com/richardktran/k8s-infrastructure.git cp ./k8s-infrastructure/nodes-setup/master.sh ./master.sh cp ./k8s-infrastructure/nodes-setup/common.sh ./common.sh cp ./k8s-infrastructure/nodes-setup/jenkins.sh ./jenkins-setup.sh sudo chmod +x common.sh sudo chmod +x master.sh sudo chmod +x jenkins-setup.sh
Run the common file to setup Docker, CRI for Docker, Kubeadm, Kubelet and Kubectl:
./common.sh
Run the setup-jenkins file to setup jenkins:
./jenkins-setup.sh
After run this command, it will show the admin password of Jenkins, access to the [MasterIP]:8080 to go to the Jenkins dashboard, enter the password to setup Jenkins.
Run the master file to init the master cluster, setup network for k8s
./master.sh
-
Install some tools in Ubuntu:
sudo apt update sudo apt install vim git curl -y
-
Clone setup files and setup K8s for master node:
git clone https://github.com/richardktran/k8s-infrastructure.git cp ./k8s-infrastructure/nodes-setup/node.sh ./node.sh cp ./k8s-infrastructure/nodes-setup/common.sh ./common.sh sudo chmod +x common.sh sudo chmod +x node.sh
Run the common file to setup Docker, CRI for Docker, Kubeadm, Kubelet and Kubectl:
./common.sh
Run the master file to init the master cluster, setup network for k8s
./node.sh
Copy the join command of master node and run it.
Example:
sudo kubeadm join 172.16.171.135:6443 --token 0piwjz.7tromi63dv28sgfo --discovery-token-ca-cert-hash sha256:ccaf5a2641b14aaa472ef0c53aa8b179ba4e5f550b8c0574f4710268205db11c --cri-socket=unix:///var/run/cri-dockerd.sock
To see the join command, go to master node and run the following command:
kubeadm token create --print-join-command
- Select create job, enter job name and select Pipeline project
-
Scroll down to the pipeline section, select the Pipeline script from SCM option. In the SCM, choose Git and enter the Repository URL (git@github.com:richardktran/k8s-infrastructure.git)
-
We will get the error, add the Credentials by add new credentials, choose SSH Username with private key, enter the username and private key.
-
To get the private key, we have to generate the ssh key in Jenkins user of master node.
sudo su jenkins mkdir .ssh cd ~/.ssh ssh-keygen -t rsa -b 4096 -C "richardktran.dev@gmail.com" cat ~/.ssh/id_rsa.pub
-
Put info of key to know_hosts
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts
-
Copy content of the id_rsa.pub file and paste to your github, copy content of the id_rsa file and paste to your credential of Jenkins.
-
The result like this:
-
-
Change the branch to build is “main” branch. Put the path to the Jenkinsfile in Script Path and click Save.
-
Setup some credential declare on Jenkinsfile.
- Get docker password from dockerhub
- Github token get from id_rsa file.
NOTE: By default, when build image. Jenkins will use jenkins network with bridge driver, we have to change to host network in Jenkinsfile on the build image stage to be able to access internet.
-
Check the current kubernetes version by run the following command:
kubectl version
-
Go to https://github.com/kubernetes/ingress-nginx/ and find the version of ingress-nginx that match with the kubernetes version.
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm search repo ingress-nginx --versions
-
Install ingress-nginx:
CHART_VERSION="4.6.1" APP_VERSION="1.7.1" helm template ingress-nginx ingress-nginx \ --repo https://kubernetes.github.io/ingress-nginx \ --version ${CHART_VERSION} \ --namespace ingress-nginx > ./nginx-ingress.${APP_VERSION}.yaml
-
Deploy the Ingress controller
kubectl create namespace ingress-nginx kubectl apply -f ./nginx-ingress.${APP_VERSION}.yaml
-
Check the port of ingress-nginx:
kubectl get svc -n ingress-nginx
If it show like this:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.97.10.71 <pending> 80:31301/TCP,443:30929/TCP 47m ingress-nginx-controller-admission ClusterIP 10.111.105.15 <none> 443/TCP 47m
It mean the port of ingress-nginx is 31301 for http and 30929 for https.
-
Open HAProxy config:
sudo vim /etc/haproxy/haproxy.cfg
-
Add the following config to the end of that file, replace the port of ingress-nginx with the port you get from step 1, replace the ip of master and worker node with your ip.
frontend http_front bind *:80 acl is_jenkins_host hdr(host) -i jenkins.richardktran.local use_backend jenkins_backend if is_jenkins_host default_backend http_back backend http_back balance roundrobin server master 172.16.171.135:31301 check server worker1 172.16.171.133:31301 check backend jenkins_backend mode http server jenkins_server 172.16.171.135:8080 check
Go to Jenkins > Manage Jenkins > System, in the "Jenkins URL" field, enter the following url:
http://jenkins.richardktran.local/
-
Validate config file & restart HAProxy:
haproxy -c -f /etc/haproxy/haproxy.cfg sudo service haproxy restart
-
In the external machine, add the following config to the end of /etc/hosts file:
172.16.171.135 richardktran.local 172.16.171.135 gateway.richardktran.local 172.16.171.135 jenkins.richardktran.local
-
Config your ingress in the value.yaml, example:
ingress: enabled: true className: "nginx" annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/proxy-body-size: "20m" nginx.ingress.kubernetes.io/limit-rps: "15" nginx.ingress.kubernetes.io/limit-rpm: "450" # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" hosts: - host: gateway.richardktran.local paths: - path: / pathType: ImplementationSpecific tls: []
-
Rebuild your helm chart and deploy it.
-
Access to the gateway.richardktran.local to see the result.
In case kubelet is not running, run the following command:
Please perform below steps on the master node. It works like charm.
1. sudo -i
2. swapoff -a
3. exit
4. strace -eopenat kubectl version