error adding container to network "cbr0": plugin type="flannel" failed (add): failed to delegate add: bridge port in error state: lower-layer-down

[bobdoah]

Since installing multus, pods are taking several minutes to be created. There are multiple error messages logged as follows:

  Warning  FailedCreatePodSandBox  29s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5a79e741217dbbd05dfe532f8346627813a11bb7ee9979e43f74b6e322042c4f": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:5a79e741217dbbd05dfe532f8346627813a11bb7ee9979e43f74b6e322042c4f Netns:/var/run/netns/cni-2739c4ea-53b0-b820-12c8-cef6ac666ee2 IfName:eth0 Args:K8S_POD_NAME=forgcc-75dbdd9547-r658c;K8S_POD_INFRA_CONTAINER_ID=5a79e741217dbbd05dfe532f8346627813a11bb7ee9979e43f74b6e322042c4f;K8S_POD_UID=976e5ba5-1551-4351-bbf4-3ad8c39ffa27;IgnoreUnknown=1;K8S_POD_NAMESPACE=apps Path: StdinData:[123 34 98 105 110 68 105 114 34 58 34 47 118 97 114 47 108 105 98 47 114 97 110 99 104 101 114 47 107 51 115 47 100 97 116 97 47 99 117 114 114 101 110 116 47 98 105 110 34 44 34 98 105 110 100 105 114 34 58 34 47 118 97 114 47 108 105 98 47 114 97 110 99 104 101 114 47 107 51 115 47 100 97 116 97 47 99 117 114 114 101 110 116 47 98 105 110 34 44 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 98 97 110 100 119 105 100 116 104 34 58 116 114 117 101 44 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 104 114 111 111 116 68 105 114 34 58 34 47 104 111 115 116 114 111 111 116 34 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 49 48 45 102 108 97 110 110 101 108 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 34 97 117 116 111 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 125]} ContainerID:"5a79e741217dbbd05dfe532f8346627813a11bb7ee9979e43f74b6e322042c4f" Netns:"/var/run/netns/cni-2739c4ea-53b0-b820-12c8-cef6ac666ee2" IfName:"eth0" Args:"K8S_POD_NAME=forgcc-75dbdd9547-r658c;K8S_POD_INFRA_CONTAINER_ID=5a79e741217dbbd05dfe532f8346627813a11bb7ee9979e43f74b6e322042c4f;K8S_POD_UID=976e5ba5-1551-4351-bbf4-3ad8c39ffa27;IgnoreUnknown=1;K8S_POD_NAMESPACE=apps" Path:"" ERRORED: error configuring pod [apps/forgcc-75dbdd9547-r658c] networking: [apps/forgcc-75dbdd9547-r658c/976e5ba5-1551-4351-bbf4-3ad8c39ffa27:cbr0]: error adding container to network "cbr0": plugin type="flannel" failed (add): failed to delegate add: bridge port in error state: lower-layer-down

Bridge interface creation seems to be stalled. I am using the manifests to install the thick plugin daemonset. I am using k3s, on Kairos/Ubuntu 22.04.3, and had to make minor tweaks to config to point it at the correct rancher cni directories:

bobdoah@slug:~% diff multus-daemonset-thick.yml k3s-homelab/infrastructure/multus-cni/multus-cni.yaml
74d73
<       - watch
114a114
>         "bindir": "/var/lib/rancher/k3s/data/current/bin",
168d167
<           terminationMessagePolicy: FallbackToLogsOnError
176c175
<               mountPath: /opt/cni/bin
---
>               mountPath: /var/lib/rancher/k3s/data/current/bin
212d210
<           terminationMessagePolicy: FallbackToLogsOnError
221c219
<             path: /etc/cni/net.d
---
>             path: /var/lib/rancher/k3s/agent/etc/cni/net.d
224c222
<             path: /opt/cni/bin
---
>             path: /var/lib/rancher/k3s/data/current/bin

These pods are not using secondary attachments, although I do have one defined.

[dougbtv]

This is an error from flannel, I think you'd be best served filing an issue with the flannel maintainers:

plugin type="flannel" failed (add): failed to delegate add: bridge port in error state: lower-layer-down