multus kubeconfig file not populated correctly after node reboot
geeky-akshay opened this issue · 2 comments
geeky-akshay commented
What happend: Apiserver IP address was empty in multus kubeconfig file
/etc/cni/net.d/multus.d/multus.kubeconfig (server: https://[])
`apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
server: https://[]:
certificate-authority-data: [snipped]==
users:
- name: multus
user:
token: "[snipped]=="
contexts:
- name: multus-context
context:
cluster: local
user: multus
current-context: multus-context`
Pods
What you expected to happen: Pods were not scheduled and the below errors were appearing in /var/log/messages
May 20 14:13:20 ace-func1-3n1 kubelet[1683]: E0520 14:13:20.399208 1683 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"d02259dca386dfd70b0d6433452e99a880b57e849f79f74993ace14d90e55fb9\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): Multus: error getting k8s client: GetK8sClient: failed to get context for the kubeconfig /etc/cni/net.d/multus.d/multus.kubeconfig: error loading config file \\\"/etc/cni/net.d/multus.d/multus.kubeconfig\\\": yaml: line 7: mapping values are not allowed in this context\"" pod="kube-system/coredns-58f4964b57-f8wkh" podUID="d4fd3ede-c09e-4a3f-abd2-8eeb6ba0b3da"
How to reproduce it (as minimally and precisely as possible): Reboot a node in multi-node Kubernetes cluster
Anything else we need to know?:
Environment:
- Multus version : ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
- Kubernetes version (use
kubectl version):
Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.1 - Primary CNI for Kubernetes cluster: Calico
- OS (e.g. from /etc/os-release):
`
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
`
- File of '/etc/cni/net.d/'
- File of '/etc/cni/multus/net.d'
- NetworkAttachment info (use
kubectl get net-attach-def -o yaml) - Target pod yaml info (with annotation, use
kubectl get pod <podname> -o yaml)
`
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2024-05-20T09:02:12Z"
generateName: kube-multus-ds-
labels:
app: multus
controller-revision-hash: 789c4467b8
name: multus
pod-template-generation: "1"
tier: node
name: kube-multus-ds-hxtkt
namespace: kube-system
ownerReferences:- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: DaemonSet
name: kube-multus-ds
uid: cc6cd4f8-540a-4593-b099-4be99d0088bd
resourceVersion: "2107922"
uid: 308c615b-8284-4c79-91f5-9c3ec976e0e6
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchFields:
- key: metadata.name
operator: In
values:
- ace-func1-3n1
containers: - args:
- --cni-version=0.3.1
- --cni-conf-dir=/host/etc/cni/net.d
- --multus-autoconfig-dir=/host/etc/cni/net.d
- --multus-log-to-stderr=true
- --multus-log-level=verbose
command: - /thin_entrypoint
image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
imagePullPolicy: IfNotPresent
name: kube-multus
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 100m
memory: 50Mi
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts: - mountPath: /host/etc/cni/net.d
name: cni - mountPath: /host/opt/cni/bin
name: cnibin - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-5wxv9
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostNetwork: true
initContainers:
- args:
- --type
- thin
command: - /install_multus
image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
imagePullPolicy: IfNotPresent
name: install-multus-binary
resources:
requests:
cpu: 10m
memory: 15Mi
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts: - mountPath: /host/opt/cni/bin
mountPropagation: Bidirectional
name: cnibin - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-5wxv9
readOnly: true
nodeName: ace-func1-3n1
preemptionPolicy: PreemptLowerPriority
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: multus
serviceAccountName: multus
terminationGracePeriodSeconds: 10
tolerations:
- effect: NoSchedule
operator: Exists - effect: NoExecute
operator: Exists - effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists - effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists - effect: NoSchedule
key: node.kubernetes.io/disk-pressure
operator: Exists - effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists - effect: NoSchedule
key: node.kubernetes.io/pid-pressure
operator: Exists - effect: NoSchedule
key: node.kubernetes.io/unschedulable
operator: Exists - effect: NoSchedule
key: node.kubernetes.io/network-unavailable
operator: Exists
volumes: - hostPath:
path: /etc/cni/net.d
type: ""
name: cni - hostPath:
path: /opt/cni/bin
type: ""
name: cnibin - name: kube-api-access-5wxv9
projected:
defaultMode: 420
sources:- serviceAccountToken:
expirationSeconds: 3607
path: token - configMap:
items:- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- key: ca.crt
- downwardAPI:
items:- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- fieldRef:
- serviceAccountToken:
- lastProbeTime: null
lastTransitionTime: "2024-05-20T09:02:13Z"
status: "True"
type: PodReadyToStartContainers - lastProbeTime: null
lastTransitionTime: "2024-05-20T09:02:14Z"
status: "True"
type: Initialized - lastProbeTime: null
lastTransitionTime: "2024-05-20T09:02:15Z"
status: "True"
type: Ready - lastProbeTime: null
lastTransitionTime: "2024-05-20T09:02:15Z"
status: "True"
type: ContainersReady - lastProbeTime: null
lastTransitionTime: "2024-05-20T09:02:12Z"
status: "True"
type: PodScheduled
containerStatuses: - containerID: containerd://fe1413443a789d09e25900c72cd3cfc2d33a6ccd5308bf75741284952ed58d18
image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760
lastState: {}
name: kube-multus
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-05-20T09:02:14Z"
hostIP: 10.52.214.11
hostIPs: - ip: 10.52.214.11
initContainerStatuses: - containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3
image: registry.rbbn.com/thirdparty/ghcr.io/k8snetworkplumbingwg/multus-cni:v4.0.2
imageID: sha256:2ce3492d2e6f598566f6ec26d08c39ca8c43df5beeefbb7ce38a1a4ef3809760
lastState: {}
name: install-multus-binary
ready: true
restartCount: 0
started: false
state:
terminated:
containerID: containerd://9e4195df464d4d98784b1d7521d5d71a775a34f334e0dc889e7f366171f8a0e3
exitCode: 0
finishedAt: "2024-05-20T09:02:13Z"
reason: Completed
startedAt: "2024-05-20T09:02:13Z"
phase: Running
podIP: 10.52.214.11
podIPs: - ip: 10.52.214.11
qosClass: Burstable
startTime: "2024-05-20T09:02:12Z"
`
- apiVersion: apps/v1
- Other log outputs (if you use multus logging)
dougbtv commented
Do you have a reproducer for this? thanks!