LXC/LXD - Kubernetes fails to load
jino-mariath opened this issue · 5 comments
I have an issue loading Kubernetes on LXD environment.
my host machine is Ubuntu 20.04
failing Kubernetes component in lxd environment.
Could you help me to fix this?
`
$ lxc profile list
+---------+----------------------------+---------+
| NAME | DESCRIPTION | USED BY |
+---------+----------------------------+---------+
| default | Default LXD profile | 0 |
+---------+----------------------------+---------+
| k8s | LXD profile for Kubernetes | 3 |
+---------+----------------------------+---------+
$ lxc profile show k8s
config:
limits.cpu: "2"
limits.memory: 2GB
limits.memory.swap: "false"
linux.kernel_modules: ip_tables,ip6_tables,nf_nat,overlay,br_netfilter
raw.lxc: "lxc.apparmor.profile=unconfined\nlxc.cap.drop= \nlxc.cgroup.devices.allow=a\nlxc.mount.auto=proc:rw
sys:rw"
security.nesting: "true"
security.privileged: "true"
description: LXD profile for Kubernetes
devices:
enp179s0f0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: k8s
used_by:
- /1.0/instances/kmaster
- /1.0/instances/kworker1
- /1.0/instances/kworker2
$
$ ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:26:10:33:a3 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp179s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.232.16.52 netmask 255.255.255.0 broadcast 10.232.16.255
inet6 fe80::735:e4df:66ac:8c7a prefixlen 64 scopeid 0x20
ether 0c:9d:92:20:47:d5 txqueuelen 1000 (Ethernet)
RX packets 5602060 bytes 6567487432 (6.5 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1029260 bytes 84934009 (84.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp179s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 0c:9d:92:20:47:d6 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 18087 bytes 1655238 (1.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18087 bytes 1655238 (1.6 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lxdbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.180.46.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::216:3eff:feb8:3230 prefixlen 64 scopeid 0x20
ether 00:16:3e:b8:32:30 txqueuelen 1000 (Ethernet)
RX packets 536636 bytes 34308840 (34.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 916693 bytes 4944232652 (4.9 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth8cf35fe0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether f6:ff:6a:c3:bd:97 txqueuelen 1000 (Ethernet)
RX packets 39639 bytes 5072701 (5.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 66630 bytes 380217792 (380.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth915b923b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 4a:68:02:9d:d7:33 txqueuelen 1000 (Ethernet)
RX packets 22021 bytes 1867814 (1.8 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 37891 bytes 192371371 (192.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vethb6f7566a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 7e:db:0d:b3:da:a1 txqueuelen 1000 (Ethernet)
RX packets 21477 bytes 1672949 (1.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 35168 bytes 191387372 (191.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
$
$./kubelx provision
Provisioning Kubernetes Cluster...
==> Bringing up kmaster
Creating kmaster
Starting kmaster
==> Running provisioner script
[TASK 1] Install containerd runtime
[TASK 2] Add apt repo for kubernetes
[TASK 3] Install Kubernetes components (kubeadm, kubelet and kubectl)
[TASK 4] Enable ssh password authentication
[TASK 5] Set root password
[TASK 6] Install additional packages
[TASK 7] Pull required containers
[TASK 8] Initialize Kubernetes Cluster
[TASK 9] Copy kube admin config to root user .kube directory
[TASK 10] Deploy Flannel network
[TASK 11] Generate and save cluster join command to /joincluster.sh
==> Bringing up kworker1
Creating kworker1
Starting kworker1
==> Running provisioner script
[TASK 1] Install containerd runtime
[TASK 2] Add apt repo for kubernetes
[TASK 3] Install Kubernetes components (kubeadm, kubelet and kubectl)
[TASK 4] Enable ssh password authentication
[TASK 5] Set root password
[TASK 6] Install additional packages
[TASK 7] Join node to Kubernetes Cluster
==> Bringing up kworker2
Creating kworker2
Starting kworker2
==> Running provisioner script
[TASK 1] Install containerd runtime
[TASK 2] Add apt repo for kubernetes
[TASK 3] Install Kubernetes components (kubeadm, kubelet and kubectl)
[TASK 4] Enable ssh password authentication
[TASK 5] Set root password
[TASK 6] Install additional packages
[TASK 7] Join node to Kubernetes Cluster
$ lxc list
+----------+---------+----------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------+---------+----------------------+------+-----------+-----------+
| kmaster | RUNNING | 10.180.46.7 (eth0) | | CONTAINER | 0 |
+----------+---------+----------------------+------+-----------+-----------+
| kworker1 | RUNNING | 10.180.46.11 (eth0) | | CONTAINER | 0 |
+----------+---------+----------------------+------+-----------+-----------+
| kworker2 | RUNNING | 10.180.46.216 (eth0) | | CONTAINER | 0 |
+----------+---------+----------------------+------+-----------+-----------+
$ lxc exec kmaster bash
root@kmaster:# kubectl get nodes# kubectl get all -n kube-system
NAME STATUS ROLES AGE VERSION
kmaster Ready control-plane,master 6m30s v1.22.0
kworker1 Ready 5m24s v1.22.0
kworker2 Ready 4m20s v1.22.0
root@kmaster:
NAME READY STATUS RESTARTS AGE
pod/coredns-78fcd69978-4wm27 0/1 ContainerCreating 0 6m19s
pod/coredns-78fcd69978-vr85t 0/1 ContainerCreating 0 6m19s
pod/etcd-kmaster 1/1 Running 0 6m35s
pod/kube-apiserver-kmaster 1/1 Running 0 6m34s
pod/kube-controller-manager-kmaster 1/1 Running 0 6m28s
pod/kube-flannel-ds-gcngr 0/1 CrashLoopBackOff 4 (17s ago) 4m27s
pod/kube-flannel-ds-mg6gn 0/1 CrashLoopBackOff 5 (20s ago) 6m19s
pod/kube-flannel-ds-r68mp 0/1 CrashLoopBackOff 4 (75s ago) 5m31s
pod/kube-proxy-72r4d 0/1 CrashLoopBackOff 5 (78s ago) 4m27s
pod/kube-proxy-hkf2c 0/1 CrashLoopBackOff 5 (2m32s ago) 5m31s
pod/kube-proxy-qd9vg 0/1 CrashLoopBackOff 6 (41s ago) 6m19s
pod/kube-scheduler-kmaster 1/1 Running 0 6m28s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 6m34s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/kube-flannel-ds 3 3 0 3 0 6m33s
daemonset.apps/kube-proxy 3 3 0 3 0 kubernetes.io/os=linux 6m34s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/coredns 0/2 2 0 6m34s
NAME DESIRED CURRENT READY AGE
replicaset.apps/coredns-78fcd69978 2 2 0 6m19s
root@kmaster:~#`
First: Why did you not follow the suggested guidelines when raising a bug?
Second: Please use code blocks - it makes it easier to read shell output.
A good bug report must include:
**Describe the bug**
**How To Reproduce**
**Expected behavior**
**Screenshots (if any)**
**Environment (please complete the following information):**
**Additional context**
Add any other context about the problem here.If you do not follow these guidelines, how on earth can you be helped? We cannot see what you are seeing so you have to help us help you and that's what these guidelines help capture.
If you do not put in any effort to clearly and concisely identify the issue, then don't expect anyone looking at your screen dumps above to put in any effort either!
Thank you, Got it, I hereby attach the screenshot, Execution, and the error for better understanding.
Execution Went fine.
Cluster Join completed successfully.
But the Kubernetes services are not fully up.
Could you help why this is not working on LXD environment?
OK - the common denominator to those issues is because flannel hasn't come up yet. What do you see in the Status when you run the following command - kubectl get nodes -o wide?
Also, have a look at the output of kubectl describe nodes kmaster - Under the Conditions section, you'll most likely see that there's a CNI issue:
Conditions on a working control-plane node
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sat, 16 Oct 2021 12:33:05 +0000 Sat, 16 Oct 2021 12:33:05 +0000 FlannelIsUp Flannel is running on this node
MemoryPressure False Sun, 17 Oct 2021 04:48:38 +0000 Sat, 16 Oct 2021 12:32:39 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 17 Oct 2021 04:48:38 +0000 Sat, 16 Oct 2021 12:32:39 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 17 Oct 2021 04:48:38 +0000 Sat, 16 Oct 2021 12:32:39 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 17 Oct 2021 04:48:38 +0000 Sat, 16 Oct 2021 12:33:11 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Also note that Flannel had some issues a few days ago that have since been resolved: flannel-io/flannel#1482
EDIT:
Actually, from that last screenshot, I see that all nodes are in Ready state - I think there's something very wrong there! DNS and CNI are critical to k8s operation. I find it odd that nodes are showing to be in Ready state under these conditions!
Let me see if I can launch an LXD environment myself.
OK - managed to launch mine with no issues. Note that I'm running Ubuntu on my laptop so LXD is not running in a VM like what you get under Vagrant. But still, if @justmeandopensource is able to do it, it ought to work for you too.
Can you check the logs for one of those containers that are crashing:
kubectl -n kube-system logs kube-flannel-ds-bb5cr -p
Thank you for your great help and explanation.
please find the details :
and Condition session :
