bento/ubuntu-22.04 vagrant box update 2 days prevent calico network to initialise
Opened this issue · 3 comments
weetat commented
Hi
After Vagrant box for bento/ubuntu-22.04 was updated https://app.vagrantup.com/bento/boxes/ubuntu-22.04 on 2 days ago, the calico network pods are not in running mode. Refer to below for more details.
Error is "Error: container create failed: pivot_root: Invalid argument".
Please advise. Thank you.
kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-658d97c59c-78lv4 1/1 Running 0 91s
kube-system calico-node-mvqw7 0/1 Init:CreateContainerError 0 91s
kube-system coredns-76f75df574-8n6wj 1/1 Running 0 91s
kube-system coredns-76f75df574-96d8g 1/1 Running 0 91s
kube-system etcd-controlplane 1/1 Running 0 104s
kube-system kube-apiserver-controlplane 1/1 Running 0 104s
kube-system kube-controller-manager-controlplane 1/1 Running 0 104s
kube-system kube-proxy-hxnk9 1/1 Running 0 91s
kube-system kube-scheduler-controlplane 1/1 Running 0 104s
kube-system metrics-server-d4dc9c4f-znwdn 0/1 Pending 0 91skubectl describe po calico-node-mvqw7 -n kube-system
Name: calico-node-mvqw7
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: calico-node
Node: controlplane/192.168.96.116
Start Time: Sat, 10 Aug 2024 00:07:49 +0000
Labels: controller-revision-hash=574c44bccd
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 192.168.96.116
IPs:
IP: 192.168.96.116
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: cri-o://bf6974f1a6c71723bf9f2a70d2c28ccf083eba89aac0e4cdb2b4ca8178aefd2e
Image: docker.io/calico/cni:v3.25.0
Image ID: docker.io/calico/cni@sha256:a38d53cb8688944eafede2f0eadc478b1b403cefeff7953da57fe9cd2d65e977
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 10 Aug 2024 00:08:02 +0000
Finished: Sat, 10 Aug 2024 00:08:02 +0000
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
install-cni:
Container ID: cri-o://4a3d1b676e31d696a440b41ff038b6c016c050c34804fbf89bdf473942416db4
Image: docker.io/calico/cni:v3.25.0
Image ID: docker.io/calico/cni@sha256:a38d53cb8688944eafede2f0eadc478b1b403cefeff7953da57fe9cd2d65e977
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 10 Aug 2024 00:08:02 +0000
Finished: Sat, 10 Aug 2024 00:08:04 +0000
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
mount-bpffs:
Container ID:
Image: docker.io/calico/node:v3.25.0
Image ID:
Port: <none>
Host Port: <none>
Command:
calico-node
-init
-best-effort
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/nodeproc from nodeproc (ro)
/sys/fs from sys-fs (rw)
/var/run/calico from var-run-calico (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.25.0
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
CALICO_IPV6POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_HEALTHENABLED: true
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/bpf from bpffs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j5hnn (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sys-fs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
bpffs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType: Directory
nodeproc:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
kube-api-access-j5hnn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 114s default-scheduler Successfully assigned kube-system/calico-node-mvqw7 to controlplane
Normal Pulling 113s kubelet Pulling image "docker.io/calico/cni:v3.25.0"
Normal Pulled 101s kubelet Successfully pulled image "docker.io/calico/cni:v3.25.0" in 11.461s (11.461s including waiting)
Normal Created 101s kubelet Created container upgrade-ipam
Normal Started 101s kubelet Started container upgrade-ipam
Normal Pulled 101s kubelet Container image "docker.io/calico/cni:v3.25.0" already present on machine
Normal Created 101s kubelet Created container install-cni
Normal Started 101s kubelet Started container install-cni
Normal Pulling 99s kubelet Pulling image "docker.io/calico/node:v3.25.0"
Normal Pulled 80s kubelet Successfully pulled image "docker.io/calico/node:v3.25.0" in 12.332s (18.852s including waiting)
Warning Failed 13s (x7 over 80s) kubelet Error: container create failed: pivot_root: Invalid argument
Normal Pulled 13s (x6 over 80s) kubelet Container image "docker.io/calico/node:v3.25.0" already present on machineAdhito commented
Just re-created my cluster and confirm this issue persist
vagrant@controlplane:~$ kubectl logs calico-node-pods -n kube-system
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-xqlsn" is waiting to start: PodInitializing
MarosBenadik commented
Yeah, same here. Interestingly, I've tried that with older versions of bento/ubuntu-22.04 (202404.23.0 and 202401.31.0) and encountered the same error.
Adhito commented
Just tried to upgrading Calico from 3.26 to 3.28 and it seemed to resolve the issue.