containers/podman

Podman errors using CNI on Risc-V archtecture

carlosedp opened this issue · 5 comments

After all work by @giuseppe towards having Podman on Risc-V architecture, everything works fine (building, running, etc) containers but I'm seeing some problems with CNI (not using --net host).

➜ podman version
Version:            1.4.4-dev
RemoteAPI Version:  1
Go Version:         devel +f980a63fcb Fri May 24 20:26:57 2019 +1000
Git Commit:         ffbc4a97801a59a887c49016a17efd0782c1aa77
Built:              Thu Jun 27 10:34:21 2019
OS/Arch:            linux/riscv64

➜ sudo podman info --debug
debug:
compiler: gc
git commit: ffbc4a97801a59a887c49016a17efd0782c1aa77
go version: devel +f980a63fcb Fri May 24 20:26:57 2019 +1000
podman version: 1.4.4-dev
host:
BuildahVersion: 1.9.0
Conmon:
    package: Unknown
    path: /usr/local/bin/conmon
    version: 'conmon version 0.4.1-dev, commit: 2a7ec7b01abd46bc3084571097bd1a949173f245'
Distribution:
    distribution: fedora
    version: "31"
MemFree: 5505134592
MemTotal: 6247546880
OCIRuntime:
    package: Unknown
    path: /usr/local/bin/crun
    version: crun 0.6
SwapFree: 0
SwapTotal: 0
arch: riscv64
cpus: 6
hostname: fedora-riscv
kernel: 5.1.0-06536-gef75bd71c5d3-dirty
os: linux
rootless: false
uptime: 12m 47.92s
registries:
blocked: null
insecure: null
search:
- docker.io
- registry.fedoraproject.org
- registry.access.redhat.com
store:
ConfigFile: /etc/containers/storage.conf
ContainerStore:
    number: 1
GraphDriverName: overlay
GraphOptions: null
GraphRoot: /var/lib/containers/storage
GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
ImageStore:
    number: 1
RunRoot: /var/run/containers/storage
VolumePath: /var/lib/containers/storage/volumes

On Fedora VM:

Linux fedora-riscv 5.1.0-06536-gef75bd71c5d3-dirty #9 SMP Mon Jun 24 18:28:34 -03 2019 riscv64 riscv64 riscv64 GNU/Linux

➜ sudo iptables --version
iptables v1.8.0 (legacy)

Starting a podman container with CNI gives:

➜ sudo podman run -d --name echo -p 8080:8080 carlosedp/echo_on_riscv
Error: unable to start container "echo": error adding firewall rules for container 85a04312ec0a55e5e93d6cd057217a7951d47493ebf9fe35cb892bab66f7e1ed: failed to add the address 10.88.0.4/32 to trusted zone: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 9 failed

After this, deleting and starting a new container with host network, the container doesn't give an error but the application becomes unaccessible until reboot.

carlosedp in ~ at fedora-riscv
➜ sudo podman run -d --name echo --net host -p 8080:8080 carlosedp/echo_on_riscv
b784d5b9ccc6c95f7867129b0a84211a5db56b4b4fa595d65959dd401f0c6c27

➜ sudo podman ps -a
CONTAINER ID  IMAGE                                     COMMAND      CREATED         STATUS             PORTS  NAMES
b784d5b9ccc6  docker.io/carlosedp/echo_on_riscv:latest  /echo-riscv  29 seconds ago  Up 27 seconds ago         echo

➜ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: No route to host

➜ sudo netstat -anp |grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      1383/echo-riscv

With host network (--net host), it works (start, stop, delete).

On Debian VM:

Linux debian-riscvqemu 5.1.0-06536-gef75bd71c5d3 #6 SMP Sun Jun 9 12:37:11 -03 2019 riscv64 GNU/Linux
➜  ~ sudo iptables --version
iptables v1.8.2 (nf_tables)

Container starts and runs with CNI but shows error when deleting.

➜  ~ sudo podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
➜  ~ sudo podman images
REPOSITORY                          TAG      IMAGE ID       CREATED      SIZE
docker.io/carlosedp/echo_on_riscv   latest   20d457ffcf56   2 days ago   9.08 MB
➜  ~ sudo podman run -d --name echo -p 8080:8080 carlosedp/echo_on_riscv
    dd905c3e6ae6f8eced9c3e870d0d54ecbb461f3da40d334623dd5cb6f0486694
➜  ~ curl localhost:8080
Hello, World! I'm running on linux/riscv64 inside a container!%
➜  ~ sudo podman rm -f echo
ERRO[0000] Error deleting network: could not teardown ipv4 dnat: running [/usr/sbin/iptables -t nat -N CNI-DN-eef0b591187e4d05dada4 --wait]: exit status 1: iptables v1.8.2 (nf_tables): Chain already exists
ERRO[0000] Error while removing pod from CNI network "podman": could not teardown ipv4 dnat: running [/usr/sbin/iptables -t nat -N CNI-DN-eef0b591187e4d05dada4 --wait]: exit status 1: iptables v1.8.2 (nf_tables): Chain already exists
ERRO[0000] unable to cleanup network for container dd905c3e6ae6f8eced9c3e870d0d54ecbb461f3da40d334623dd5cb6f0486694: "error tearing down CNI namespace configuration for container dd905c3e6ae6f8eced9c3e870d0d54ecbb461f3da40d334623dd5cb6f0486694: could not teardown ipv4 dnat: running [/usr/sbin/iptables -t nat -N CNI-DN-eef0b591187e4d05dada4 --wait]: exit status 1: iptables v1.8.2 (nf_tables): Chain already exists\n"
dd905c3e6ae6f8eced9c3e870d0d54ecbb461f3da40d334623dd5cb6f0486694

After this, even starting a new container with host network, the application becomes unaccessible until reboot.

➜  ~ sudo podman run -d --net host --name echo -p 8080:8080 carlosedp/echo_on_riscv
f5c55947ec0a54bf1e26676f9781c8149967278a7df8f328e712313be8f8752f
➜  ~ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused

Its port only gets ipv6 stack:

➜  ~ sudo podman ps -a
CONTAINER ID  IMAGE                                     COMMAND      CREATED         STATUS             PORTS  NAMES
f5c55947ec0a  docker.io/carlosedp/echo_on_riscv:latest  /echo-riscv  33 seconds ago  Up 31 seconds ago         echo
➜  ~ sudo netstat -anp |grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      1252/echo-riscv

Let me know how can I further help debugging this. There is a Debian VM available for download here in case needed.

could you try without nftables?

When I reverted the iptables to legacy (on Debian), it worked but printed tons of messages to dmesg.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

➜  ~ sudo iptables --version
iptables v1.8.2 (nf_tables)

➜  ~ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
➜  ~ sudo iptables --version
iptables v1.8.2 (legacy)

➜  ~ sudo podman run -d --name echo -p 8080:8080 carlosedp/echo_on_riscv
e9d2712b55c2d3c508d23715f08bf3f9b927f2ed11a068b1470f0a1f07acfbfe
➜  ~ sudo podman ps -a
CONTAINER ID  IMAGE                                     COMMAND      CREATED         STATUS            PORTS                   NAMES
e9d2712b55c2  docker.io/carlosedp/echo_on_riscv:latest  /echo-riscv  32 seconds ago  Up 6 seconds ago  0.0.0.0:8080->8080/tcp  echo
➜  ~ curl localhost:8080
Hello, World! I'm running on linux/riscv64 inside a container!%                                                                                                                          

➜  ~ sudo podman rm -f echo
e9d2712b55c2d3c508d23715f08bf3f9b927f2ed11a068b1470f0a1f07acfbfe
➜  ~ sudo podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

➜  ~ uname -a
Linux debian-riscvqemu 5.2.0-rc7-dirty #1 SMP Mon Jul 1 15:23:18 -03 2019 riscv64 GNU/Linux

➜  ~ sudo lsmod
Module                  Size  Used by
veth                   21403  0
overlay               136107  0

Dmesg output:

IPVS: ftp: loaded support on port[0] = 21
[  482.999318] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  483.008785] cni0: port 1(veth3146e1eb) entered blocking state
[  483.009208] cni0: port 1(veth3146e1eb) entered disabled state
[  483.011144] device veth3146e1eb entered promiscuous mode
[  483.013110] cni0: port 1(veth3146e1eb) entered blocking state
[  483.013401] cni0: port 1(veth3146e1eb) entered forwarding state
[  483.538207] bpfilter: Loaded bpfilter_umh pid 1019
[  483.541646] bpfilter: read fail 0
[  483.618109] bpfilter: Loaded bpfilter_umh pid 1021
[  483.620639] bpfilter: read fail 0
[  483.724909] bpfilter: Loaded bpfilter_umh pid 1024
[  483.727379] bpfilter: read fail 0
[  483.805999] bpfilter: Loaded bpfilter_umh pid 1026
[  483.809978] bpfilter: read fail 0
[  483.882854] bpfilter: Loaded bpfilter_umh pid 1028
[  483.888180] bpfilter: read fail 0
[  483.960195] bpfilter: Loaded bpfilter_umh pid 1030
[  483.962496] bpfilter: read fail 0
[  484.070211] bpfilter: Loaded bpfilter_umh pid 1033
[  484.072945] bpfilter: read fail 0
[  484.156220] bpfilter: Loaded bpfilter_umh pid 1035
[  484.158713] bpfilter: read fail 0
[  484.234698] bpfilter: Loaded bpfilter_umh pid 1037
[  484.238543] bpfilter: read fail 0
[  484.334619] bpfilter: Loaded bpfilter_umh pid 1040
[  484.337187] bpfilter: read fail 0
[  484.414064] bpfilter: Loaded bpfilter_umh pid 1042
[  484.417153] bpfilter: read fail 0
[  484.486426] bpfilter: Loaded bpfilter_umh pid 1044
[  484.489146] bpfilter: read fail 0
[  484.566380] bpfilter: Loaded bpfilter_umh pid 1046
[  484.569231] bpfilter: read fail 0
[  484.645488] bpfilter: Loaded bpfilter_umh pid 1048
[  484.647778] bpfilter: read fail 0
[  484.749962] bpfilter: Loaded bpfilter_umh pid 1051
[  484.754669] bpfilter: read fail 0
[  484.833332] bpfilter: Loaded bpfilter_umh pid 1053
[  484.835951] bpfilter: read fail 0
[  484.902650] bpfilter: Loaded bpfilter_umh pid 1055
[  484.905421] bpfilter: read fail 0
[  484.980342] bpfilter: Loaded bpfilter_umh pid 1057
[  484.983461] bpfilter: read fail 0
[  485.081306] bpfilter: Loaded bpfilter_umh pid 1060
[  485.083398] bpfilter: read fail 0
[  485.163044] bpfilter: Loaded bpfilter_umh pid 1062
[  485.167054] bpfilter: read fail 0
[  485.238150] bpfilter: Loaded bpfilter_umh pid 1064
[  485.241141] bpfilter: read fail 0
[  485.331459] bpfilter: Loaded bpfilter_umh pid 1066
[  485.333979] bpfilter: read fail 0
[  485.418459] bpfilter: Loaded bpfilter_umh pid 1068
[  485.421860] bpfilter: read fail 0
[  485.492946] bpfilter: Loaded bpfilter_umh pid 1070
[  485.495238] bpfilter: read fail 0
[  485.595622] bpfilter: Loaded bpfilter_umh pid 1073
[  485.598795] bpfilter: read fail 0
[  485.674126] bpfilter: Loaded bpfilter_umh pid 1075
[  485.677535] bpfilter: read fail 0
[  485.747706] bpfilter: Loaded bpfilter_umh pid 1077
[  485.750403] bpfilter: read fail 0
[  485.846976] bpfilter: Loaded bpfilter_umh pid 1080
[  485.849712] bpfilter: read fail 0
[  485.935093] bpfilter: Loaded bpfilter_umh pid 1082
[  485.937932] bpfilter: read fail 0
[  486.005634] bpfilter: Loaded bpfilter_umh pid 1084
[  486.007922] bpfilter: read fail 0
[  486.082394] bpfilter: Loaded bpfilter_umh pid 1086
[  486.086336] bpfilter: read fail 0
[  486.167131] bpfilter: Loaded bpfilter_umh pid 1088
[  486.171281] bpfilter: read fail 0
[  486.965456] bpfilter: Loaded bpfilter_umh pid 1098
[  486.967699] bpfilter: read fail 0
[  487.045082] bpfilter: Loaded bpfilter_umh pid 1100
[  487.047509] bpfilter: read fail 0
[  487.125805] bpfilter: Loaded bpfilter_umh pid 1102
[  487.128777] bpfilter: read fail 0
[  487.210287] bpfilter: Loaded bpfilter_umh pid 1104
[  487.212839] bpfilter: read fail 0
[  487.308899] bpfilter: Loaded bpfilter_umh pid 1107
[  487.311002] bpfilter: read fail 0
[  487.389674] bpfilter: Loaded bpfilter_umh pid 1109
[  487.393822] bpfilter: read fail 0
[  487.470507] bpfilter: Loaded bpfilter_umh pid 1111
[  487.472818] bpfilter: read fail 0
[  487.542661] bpfilter: Loaded bpfilter_umh pid 1113
[  487.545453] bpfilter: read fail 0
[  487.654284] bpfilter: Loaded bpfilter_umh pid 1116
[  487.656976] bpfilter: read fail 0
[  487.741475] bpfilter: Loaded bpfilter_umh pid 1118
[  487.744020] bpfilter: read fail 0
[  487.821101] bpfilter: Loaded bpfilter_umh pid 1120
[  487.823236] bpfilter: read fail 0
[  487.895456] bpfilter: Loaded bpfilter_umh pid 1122
[  487.900855] bpfilter: read fail 0
[  487.982815] bpfilter: Loaded bpfilter_umh pid 1124
[  487.986507] bpfilter: read fail 0
[  488.056531] bpfilter: Loaded bpfilter_umh pid 1126
[  488.059848] bpfilter: read fail 0
[  488.128938] bpfilter: Loaded bpfilter_umh pid 1128
[  488.131313] bpfilter: read fail 0
[  488.210989] bpfilter: Loaded bpfilter_umh pid 1130
[  488.213630] bpfilter: read fail 0
[  488.313462] bpfilter: Loaded bpfilter_umh pid 1133
[  488.315565] bpfilter: read fail 0
[  488.389689] bpfilter: Loaded bpfilter_umh pid 1135
[  488.392668] bpfilter: read fail 0
[  488.476447] bpfilter: Loaded bpfilter_umh pid 1137
[  488.479979] bpfilter: read fail 0
[  488.546801] bpfilter: Loaded bpfilter_umh pid 1139
[  488.549797] bpfilter: read fail 0
[  488.618770] bpfilter: Loaded bpfilter_umh pid 1141
[  488.622351] bpfilter: read fail 0
[  488.704279] bpfilter: Loaded bpfilter_umh pid 1143
[  488.708683] bpfilter: read fail 0
[  488.777704] bpfilter: Loaded bpfilter_umh pid 1145
[  488.780309] bpfilter: read fail 0
[  488.865221] bpfilter: Loaded bpfilter_umh pid 1147
[  488.867544] bpfilter: read fail 0
[  488.942777] bpfilter: Loaded bpfilter_umh pid 1149
[  488.945598] bpfilter: read fail 0
[  489.022330] bpfilter: Loaded bpfilter_umh pid 1151
[  489.026666] bpfilter: read fail 0
[  489.117361] bpfilter: Loaded bpfilter_umh pid 1154
[  489.120734] bpfilter: read fail 0
[  489.200329] bpfilter: Loaded bpfilter_umh pid 1156
[  489.202532] bpfilter: read fail 0
[  489.280181] bpfilter: Loaded bpfilter_umh pid 1158
[  489.282418] bpfilter: read fail 0
[  489.360973] bpfilter: Loaded bpfilter_umh pid 1160
[  489.363299] bpfilter: read fail 0
[  489.440289] bpfilter: Loaded bpfilter_umh pid 1162
[  489.442796] bpfilter: read fail 0
[  489.514107] bpfilter: Loaded bpfilter_umh pid 1164
[  489.517271] bpfilter: read fail 0
[  489.601588] bpfilter: Loaded bpfilter_umh pid 1166
[  489.604953] bpfilter: read fail 0
[  489.676999] bpfilter: Loaded bpfilter_umh pid 1168
[  489.679379] bpfilter: read fail 0
[  489.761773] bpfilter: Loaded bpfilter_umh pid 1170
[  489.764757] bpfilter: read fail 0
[  489.866316] bpfilter: Loaded bpfilter_umh pid 1173
[  489.869240] bpfilter: read fail 0
[  489.947304] bpfilter: Loaded bpfilter_umh pid 1175
[  489.950116] bpfilter: read fail 0
[  490.023831] bpfilter: Loaded bpfilter_umh pid 1177
[  490.026753] bpfilter: read fail 0
[  490.102438] bpfilter: Loaded bpfilter_umh pid 1179
[  490.104981] bpfilter: read fail 0
[  490.201666] bpfilter: Loaded bpfilter_umh pid 1182
[  490.205027] bpfilter: read fail 0
[  490.291370] bpfilter: Loaded bpfilter_umh pid 1184
[  490.294742] bpfilter: read fail 0
[  490.371100] bpfilter: Loaded bpfilter_umh pid 1186
[  490.374279] bpfilter: read fail 0
[  490.452492] bpfilter: Loaded bpfilter_umh pid 1188
[  490.454933] bpfilter: read fail 0
[  490.529454] bpfilter: Loaded bpfilter_umh pid 1190
[  490.531884] bpfilter: read fail 0
[  490.610968] bpfilter: Loaded bpfilter_umh pid 1192
[  490.613576] bpfilter: read fail 0
[  490.728894] bpfilter: Loaded bpfilter_umh pid 1195
[  490.731175] bpfilter: read fail 0
[  490.810203] bpfilter: Loaded bpfilter_umh pid 1197
[  490.813604] bpfilter: read fail 0
[  490.887244] bpfilter: Loaded bpfilter_umh pid 1199
[  490.890270] bpfilter: read fail 0
[  490.962803] bpfilter: Loaded bpfilter_umh pid 1201
[  490.966042] bpfilter: read fail 0
[  491.041809] bpfilter: Loaded bpfilter_umh pid 1203
[  491.046517] bpfilter: read fail 0
[  491.117631] bpfilter: Loaded bpfilter_umh pid 1205
[  491.121083] bpfilter: read fail 0
[  491.202797] bpfilter: Loaded bpfilter_umh pid 1207
[  491.205954] bpfilter: read fail 0
[  491.283040] bpfilter: Loaded bpfilter_umh pid 1209
[  491.285852] bpfilter: read fail 0
[  491.381742] bpfilter: Loaded bpfilter_umh pid 1212
[  491.385921] bpfilter: read fail 0
[  491.465392] bpfilter: Loaded bpfilter_umh pid 1214
[  491.468914] bpfilter: read fail 0
[  491.547025] bpfilter: Loaded bpfilter_umh pid 1216
[  491.549871] bpfilter: read fail 0
[  491.652851] bpfilter: Loaded bpfilter_umh pid 1219
[  491.657464] bpfilter: read fail 0
[  491.732454] bpfilter: Loaded bpfilter_umh pid 1221
[  491.738456] bpfilter: read fail 0
[  491.817739] bpfilter: Loaded bpfilter_umh pid 1223
[  491.821157] bpfilter: read fail 0
[  491.904940] bpfilter: Loaded bpfilter_umh pid 1225
[  491.908796] bpfilter: read fail 0
[  491.978127] bpfilter: Loaded bpfilter_umh pid 1227
[  491.982644] bpfilter: read fail 0
[  492.091435] bpfilter: Loaded bpfilter_umh pid 1230
[  492.095137] bpfilter: read fail 0
[  492.180617] bpfilter: Loaded bpfilter_umh pid 1232
[  492.183447] bpfilter: read fail 0
[  492.270872] bpfilter: Loaded bpfilter_umh pid 1234
[  492.275170] bpfilter: read fail 0
[  492.361116] bpfilter: Loaded bpfilter_umh pid 1236
[  492.365222] bpfilter: read fail 0
[  492.442867] bpfilter: Loaded bpfilter_umh pid 1238
[  492.446347] bpfilter: read fail 0
[  492.520594] bpfilter: Loaded bpfilter_umh pid 1240
[  492.523109] bpfilter: read fail 0
[  492.590409] bpfilter: Loaded bpfilter_umh pid 1242
[  492.594855] bpfilter: read fail 0
[  492.665398] bpfilter: Loaded bpfilter_umh pid 1244
[  492.670966] bpfilter: read fail 0
[  492.738321] bpfilter: Loaded bpfilter_umh pid 1246
[  492.742471] bpfilter: read fail 0
[  492.825439] bpfilter: Loaded bpfilter_umh pid 1248
[  492.830920] bpfilter: read fail 0
[  492.913423] bpfilter: Loaded bpfilter_umh pid 1250
[  492.918659] bpfilter: read fail 0
[  492.990045] bpfilter: Loaded bpfilter_umh pid 1252
[  492.993191] bpfilter: read fail 0
[  493.095773] bpfilter: Loaded bpfilter_umh pid 1255
[  493.099231] bpfilter: read fail 0
[  493.172091] bpfilter: Loaded bpfilter_umh pid 1257
[  493.174883] bpfilter: read fail 0
[  493.248863] bpfilter: Loaded bpfilter_umh pid 1259
[  493.251290] bpfilter: read fail 0
[  493.330174] bpfilter: Loaded bpfilter_umh pid 1262
[  493.332992] bpfilter: read fail 0
[  493.437644] bpfilter: Loaded bpfilter_umh pid 1265
[  493.442533] bpfilter: read fail 0
[  493.520644] bpfilter: Loaded bpfilter_umh pid 1267
[  493.525061] bpfilter: read fail 0
[  493.600074] bpfilter: Loaded bpfilter_umh pid 1269
[  493.601735] bpfilter: read fail 0
[  493.675430] bpfilter: Loaded bpfilter_umh pid 1271
[  493.678717] bpfilter: read fail 0
[  493.750451] bpfilter: Loaded bpfilter_umh pid 1273
[  493.755349] bpfilter: read fail 0
[  493.853962] bpfilter: Loaded bpfilter_umh pid 1276
[  493.856453] bpfilter: read fail 0
[  493.941903] bpfilter: Loaded bpfilter_umh pid 1278
[  493.944961] bpfilter: read fail 0
[  494.021776] bpfilter: Loaded bpfilter_umh pid 1280
[  494.024487] bpfilter: read fail 0
[  494.105606] bpfilter: Loaded bpfilter_umh pid 1282
[  494.107955] bpfilter: read fail 0
[  494.187849] bpfilter: Loaded bpfilter_umh pid 1284
[  494.190672] bpfilter: read fail 0
[  494.264126] bpfilter: Loaded bpfilter_umh pid 1286
[  494.266543] bpfilter: read fail 0
[  494.342018] bpfilter: Loaded bpfilter_umh pid 1288
[  494.344611] bpfilter: read fail 0
[  494.446467] bpfilter: Loaded bpfilter_umh pid 1291
[  494.449080] bpfilter: read fail 0
[  494.523320] bpfilter: Loaded bpfilter_umh pid 1293
[  494.525871] bpfilter: read fail 0
[  494.602081] bpfilter: Loaded bpfilter_umh pid 1295
[  494.605125] bpfilter: read fail 0
[  494.685536] bpfilter: Loaded bpfilter_umh pid 1297
[  494.691487] bpfilter: read fail 0
[  494.766004] bpfilter: Loaded bpfilter_umh pid 1299
[  494.768844] bpfilter: read fail 0
[  494.874364] bpfilter: Loaded bpfilter_umh pid 1302
[  494.877009] bpfilter: read fail 0
[  494.962840] bpfilter: Loaded bpfilter_umh pid 1304
[  494.966222] bpfilter: read fail 0
[  495.037308] bpfilter: Loaded bpfilter_umh pid 1306
[  495.041641] bpfilter: read fail 0
[  495.121349] bpfilter: Loaded bpfilter_umh pid 1308
[  495.123662] bpfilter: read fail 0
[  495.205330] bpfilter: Loaded bpfilter_umh pid 1310
[  495.208572] bpfilter: read fail 0
[  495.286329] bpfilter: Loaded bpfilter_umh pid 1312
[  495.289443] bpfilter: read fail 0
[  495.362533] bpfilter: Loaded bpfilter_umh pid 1314
[  495.365081] bpfilter: read fail 0
[  495.469907] bpfilter: Loaded bpfilter_umh pid 1317
[  495.472365] bpfilter: read fail 0
[  495.551999] bpfilter: Loaded bpfilter_umh pid 1319
[  495.554397] bpfilter: read fail 0
[  495.632490] bpfilter: Loaded bpfilter_umh pid 1321
[  495.635604] bpfilter: read fail 0
[  495.706544] bpfilter: Loaded bpfilter_umh pid 1323
[  495.710185] bpfilter: read fail 0
[  495.779144] bpfilter: Loaded bpfilter_umh pid 1325
[  495.782160] bpfilter: read fail 0
[  495.861924] bpfilter: Loaded bpfilter_umh pid 1327
[  495.864865] bpfilter: read fail 0
[  495.942338] bpfilter: Loaded bpfilter_umh pid 1329
[  495.946331] bpfilter: read fail 0
[  496.030446] bpfilter: Loaded bpfilter_umh pid 1331
[  496.034078] bpfilter: read fail 0
[  496.103749] bpfilter: Loaded bpfilter_umh pid 1333
[  496.106104] bpfilter: read fail 0
[  496.182427] bpfilter: Loaded bpfilter_umh pid 1335
[  496.186238] bpfilter: read fail 0
[  496.265433] bpfilter: Loaded bpfilter_umh pid 1337
[  496.270087] bpfilter: read fail 0
[  496.342414] bpfilter: Loaded bpfilter_umh pid 1339
[  496.346280] bpfilter: read fail 0
[  496.422891] bpfilter: Loaded bpfilter_umh pid 1341
[  496.426543] bpfilter: read fail 0
[  496.502951] bpfilter: Loaded bpfilter_umh pid 1343
[  496.507400] bpfilter: read fail 0
[  496.587587] bpfilter: Loaded bpfilter_umh pid 1345
[  496.590607] bpfilter: read fail 0
[  496.689113] bpfilter: Loaded bpfilter_umh pid 1348
[  496.691356] bpfilter: read fail 0
[  496.766414] bpfilter: Loaded bpfilter_umh pid 1350
[  496.769878] bpfilter: read fail 0
[  496.841549] bpfilter: Loaded bpfilter_umh pid 1352
[  496.844712] bpfilter: read fail 0
[  496.918403] bpfilter: Loaded bpfilter_umh pid 1354
[  496.921607] bpfilter: read fail 0
[  497.026067] bpfilter: Loaded bpfilter_umh pid 1357
[  497.028763] bpfilter: read fail 0
[  497.107006] bpfilter: Loaded bpfilter_umh pid 1359
[  497.112759] bpfilter: read fail 0
[  497.180511] bpfilter: Loaded bpfilter_umh pid 1361
[  497.186653] bpfilter: read fail 0
[  497.294096] bpfilter: Loaded bpfilter_umh pid 1364
[  497.297103] bpfilter: read fail 0
[  497.382024] bpfilter: Loaded bpfilter_umh pid 1366
[  497.386359] bpfilter: read fail 0
[  497.453985] bpfilter: Loaded bpfilter_umh pid 1368
[  497.456677] bpfilter: read fail 0
[  497.533877] bpfilter: Loaded bpfilter_umh pid 1370
[  497.535895] bpfilter: read fail 0
[  497.604800] bpfilter: Loaded bpfilter_umh pid 1372
[  497.607026] bpfilter: read fail 0
[  497.708776] bpfilter: Loaded bpfilter_umh pid 1375
[  497.713541] bpfilter: read fail 0
[  497.785366] bpfilter: Loaded bpfilter_umh pid 1377
[  497.790133] bpfilter: read fail 0
[  497.862502] bpfilter: Loaded bpfilter_umh pid 1379
[  497.865896] bpfilter: read fail 0
[  497.969221] bpfilter: Loaded bpfilter_umh pid 1382
[  497.973521] bpfilter: read fail 0
[  498.047118] bpfilter: Loaded bpfilter_umh pid 1384
[  498.050717] bpfilter: read fail 0
[  498.122645] bpfilter: Loaded bpfilter_umh pid 1386
[  498.127103] bpfilter: read fail 0
[  498.207098] bpfilter: Loaded bpfilter_umh pid 1388
[  498.209509] bpfilter: read fail 0
[  498.283018] bpfilter: Loaded bpfilter_umh pid 1390
[  498.286161] bpfilter: read fail 0
[  498.393844] bpfilter: Loaded bpfilter_umh pid 1393
[  498.396578] bpfilter: read fail 0
[  498.482387] bpfilter: Loaded bpfilter_umh pid 1395
[  498.484791] bpfilter: read fail 0
[  498.567703] bpfilter: Loaded bpfilter_umh pid 1397
[  498.570762] bpfilter: read fail 0
[  498.647906] bpfilter: Loaded bpfilter_umh pid 1399
[  498.651078] bpfilter: read fail 0
[  498.736907] bpfilter: Loaded bpfilter_umh pid 1401
[  498.738281] bpfilter: read fail 0
[  498.811660] bpfilter: Loaded bpfilter_umh pid 1403
[  498.814666] bpfilter: read fail 0
[  498.925133] bpfilter: Loaded bpfilter_umh pid 1406
[  498.928632] bpfilter: read fail 0
[  499.000769] bpfilter: Loaded bpfilter_umh pid 1408
[  499.004027] bpfilter: read fail 0
[  499.082029] bpfilter: Loaded bpfilter_umh pid 1410
[  499.084383] bpfilter: read fail 0
[  499.160444] bpfilter: Loaded bpfilter_umh pid 1412
[  499.164745] bpfilter: read fail 0
[  499.237266] bpfilter: Loaded bpfilter_umh pid 1414
[  499.239425] bpfilter: read fail 0
[  499.310150] bpfilter: Loaded bpfilter_umh pid 1416
[  499.312866] bpfilter: read fail 0
[  499.397355] bpfilter: Loaded bpfilter_umh pid 1418
[  499.399995] bpfilter: read fail 0
[  499.477655] bpfilter: Loaded bpfilter_umh pid 1420
[  499.481946] bpfilter: read fail 0
[  499.573026] bpfilter: Loaded bpfilter_umh pid 1423
[  499.577940] bpfilter: read fail 0
[  499.652667] bpfilter: Loaded bpfilter_umh pid 1425
[  499.654803] bpfilter: read fail 0
[  499.732544] bpfilter: Loaded bpfilter_umh pid 1427
[  499.735251] bpfilter: read fail 0
[  499.818444] bpfilter: Loaded bpfilter_umh pid 1429
[  499.821679] bpfilter: read fail 0
[  499.899098] bpfilter: Loaded bpfilter_umh pid 1431
[  499.904875] bpfilter: read fail 0
[  499.980609] bpfilter: Loaded bpfilter_umh pid 1433
[  499.983053] bpfilter: read fail 0
[  500.077906] bpfilter: Loaded bpfilter_umh pid 1436
[  500.078351] bpfilter: write fail -32
[  533.755966] cni0: port 1(veth3146e1eb) entered disabled state
[  533.760370] device veth3146e1eb left promiscuous mode
[  533.760590] cni0: port 1(veth3146e1eb) entered disabled state

Both VMs, Debian and Fedora uses exactly the same Kernel and modules but on Fedora (that iptables show as legacy) I get the error. My Kernel config is on https://gist.github.com/carlosedp/4100815296675d126334f8f6afba4e93.

@dcbw do you know what could be causing this issue?

Ping @dcbw @giuseppe any idea on this? Thanks!

After rebuilding the plugins from https://github.com/containernetworking/plugins and placing them on /opt/cni/bin, containers started with CNI networking works perfectly.
Closing this.