Network hotplug issues with firecracker
networkop opened this issue · 7 comments
I'm trying to test the network hotplug functionality with kata-fc with netmon enabled. When adding a new interface to the container's namespace, it's not showing up inside the VM. At the same time the following can be observed in the logs:
May 03 10:27:12 xps kata-runtime[300067]: time="2021-05-03T10:27:12.948107948+01:00" level=warning msg="hotplugAddDevice: unsupported device" arch=amd64 command=kata-network container=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 devInfo="&{{{85daa5b4-adcc-44bc-9fd0-18891843ea6b br1_kata {tap1_kata 52:8f:75:ca:29:11 []} [] []} {eth1 46:83:0c:45:88:df []} 2} {{{0 65000 0 eth1 52:8f:75:ca:29:11 0 0 0 0 <nil> <nil> 0 <nil> <nil> unknown 0 0 0 0 0 []} veth} [] [] {[] [] []} []} virtual }" deviceType=2 name=kata-runtime pid=300067 sandbox=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 source=virtcontainers subsystem=firecracker
May 03 10:27:12 xps kata-runtime[300067]: time="2021-05-03T10:27:12.948250174+01:00" level=error msg="Error attach virtual ep" arch=amd64 command=kata-network container=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 error="Could not hot add device: unsupported device: &{{{85daa5b4-adcc-44bc-9fd0-18891843ea6b br1_kata {tap1_kata 52:8f:75:ca:29:11 []} [] []} {eth1 46:83:0c:45:88:df []} 2} {{{0 65000 0 eth1 52:8f:75:ca:29:11 0 0 0 0 <nil> <nil> 0 <nil> <nil> unknown 0 0 0 0 0 []} veth} [] [] {[] [] []} []} virtual }, type: 2" name=kata-runtime pid=300067 sandbox=b4b374a2855da7b1e2353ba80bb21ef9091e9519e4253a17dc5d91b7a7577af0 source=virtcontainers subsystem=network
Inside the conainer's namespace, I can see both the new interface (eth1) and that corresponding tap interface (tap1_kata)
p -br link show
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
tap0_kata UP fa:4a:06:3e:48:d7 <BROADCAST,MULTICAST,UP,LOWER_UP>
tap1_kata DOWN ce:9d:9a:bd:24:fa <NO-CARRIER,BROADCAST,MULTICAST,UP>
eth0@if427 UP 02:42:ac:14:14:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
eth1@if431 UP ae:a6:4b:71:35:4b <BROADCAST,MULTICAST,UP,LOWER_UP>
The same functionality works with QEMU VMM just fine. Is this supposed to work with FC and/or CLH?
When running with CLH as the VMM, the error log looks like this:
May 03 11:00:48 xps kata-runtime[307307]: time="2021-05-03T11:00:48.402362277+01:00" level=error msg="add interface failed" arch=amd64 command=kata-network container=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 error="cannot hotplug device: unsupported device type '2'" name=kata-runtime pid=307307 resulting-interface="<nil>" sandbox=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 source=runtime
May 03 11:00:48 xps kata-runtime[307307]: time="2021-05-03T11:00:48.402420499+01:00" level=error msg="cannot hotplug device: unsupported device type '2'" arch=amd64 command=kata-network container=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 name=kata-runtime pid=307307 sandbox=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 source=runtime
May 03 11:00:48 xps kata-netmon[306894]: time="2021-05-03T11:00:48.404259786+01:00" level=fatal msg="handleEvents()" error="exit status 1" name=kata-netmon pid=306894 sandbox=dc15d5bb0e3a3d9274227d329e818d2039a3f4c4f07d26a8c0d4b294e595ea72 source=netmon
Looks like CLH doesn't support hotplugging of network devices:
Line 471 in 59e2273
So, I guess this is by design, so feel free to close this issue.
I'm just curious if it's possible to avoid hotplugging and connect multiple network interfaces before FC VM starts.. Is this possible?
It’s been a while… but I believe you can create a runc container, add a second interface to it, and then create a kata container that uses the net of the first runc container. This’ll result in a kata container running with two interfaces at boot, rather than relying in hotplug.