docker network rm messes up iptables
diamondT opened this issue · 10 comments
Description
When removing a network, either via docker network rm or via a docker compose down command, iptables rules are not updated correctly. Seems that rules belonging to other networks (including the default) are also being removed.
As a consequence, containers on the default (or possibly any) network cannot access the internet and inter-network communication is lost.
Restarting the daemon does bring back the rules for the default network but whenever a network is removed, things break again.
Reproduce
- restart daemon:
systemctl restart docker - observe iptables:
iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
- create a network:
docker network create my-net - remove network:
docker network rm my-net - observe iptables:
iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
The entries for the default network (docker0) have disappeared.
6. try to access the internet from within a container attached to the default network:
docker run --rm alpine ping -c 4 172.217.17.228
PING 172.217.17.228 (172.217.17.228): 56 data bytes
--- 172.217.17.228 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
(pinging www.google.com)
Expected behavior
Removing a network should not affect the default or other networks.
docker version
Client:
Version: 26.1.5-ce
API version: 1.45
Go version: go1.21.13
Git commit: 411e817ddf71
Built: Tue Nov 12 06:34:28 2024
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 26.1.5-ce
API version: 1.45 (minimum version 1.24)
Go version: go1.21.13
Git commit: 411e817ddf71
Built: Tue Nov 12 06:34:28 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.23
GitCommit: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb363254b69
docker-init:
Version: 0.2.0_catatonit
GitCommit:docker info
Client:
Version: 26.1.5-ce
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.17.1
Path: /usr/lib/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: 2.30.3
Path: /usr/lib/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 77
Server Version: 26.1.5-ce
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 oci runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc version: v1.2.2-0-g7cb363254b69
init version:
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.11.8-1-default
Operating System: openSUSE Tumbleweed
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.24GiB
Name: nefeli
ID: DN5K:YLAJ:3QJL:BKMM:FZS7:AA22:H6VJ:4CJ7:7LUH:MAXW:D4T6:S5IK
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
No response
Hi @diamondT - thank you for reporting this ... I've tried to repro (using 26.1.5-ce on SUSE), but for me it behaves as expected - only rules for the disconnected network are removed.
Please could you re-raise this in https://github.com/moby/moby/issues - and include debug logs that cover start up (creation of docker0 rules), creation of the other network, and its removal ... at debug level there should be log lines showing the iptables commands.
(Unfortunately it's not possible to move issues between projects, and this doesn't sound like a CLI issue.)
hi @robmry,
thanks for taking the time to reply and for the hint about the debug logs.
indeed i'm not seeing anything suspicious in the logs, all the iptables commands seem to be targeting the correct network.
maybe something else in the system messes up the iptables rules and not docker...
closing this and if needed i'll follow up with the moby project as advised.
thanks,
Thanks @diamondT ... there are some notes here about tools that tend to interfere with docker networking (dnsmasq, netscript, netplan), and how to "un-manage" docker interfaces - https://docs.docker.com/engine/daemon/troubleshoot/#networking
If you find another one, do let us know, and we can add it to the docs.
Faced the same problem after a recent update
docker version
Client:
Version: 26.1.5-ce
API version: 1.45
Go version: go1.21.13
Git commit: 411e817ddf71
Built: Tue Nov 12 06:34:28 2024
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 26.1.5-ce
API version: 1.45 (minimum version 1.24)
Go version: go1.21.13
Git commit: 411e817ddf71
Built: Tue Nov 12 06:34:28 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.23
GitCommit: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb363254b69
docker-init:
Version: 0.2.0_catatonit
GitCommit:
docker info
Client:
Version: 26.1.5-ce
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.17.1
Path: /usr/lib/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: 2.30.3
Path: /usr/lib/docker/cli-plugins/docker-compose
Server:
Containers: 8
Running: 0
Paused: 0
Stopped: 8
Images: 11
Server Version: 26.1.5-ce
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 oci runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc version: v1.2.2-0-g7cb363254b69
init version:
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.11.8-1-default
Operating System: openSUSE Tumbleweed
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.11GiB
Name: 3v3n7-h0r1z0n
ID: f929c7d7-a0cd-4f7c-a8fd-3c49226433fb
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Faced the same problem after a recent update
Hi @hz61p1 ... upgrade of the OS, or docker? Do you know which versions?
Faced the same problem after a recent update
Hi @hz61p1 ... upgrade of the OS, or docker? Do you know which versions?
After a complete OS upgrade. The updated packages included docker and iptables packages
still haven't found out what's wrong but here what - i think - is happening: for some reason, iptables won't report correctly on matching rules with interface names.
doesn't look like a docker issue, but in any case this is how the story goes:
the iptables version in my system is
$> iptables --version
iptables v1.8.11 (nf_tables)
when creating a new new docker network, dockerd is first checking if the rule exists with iptables -C and then inserts the rule with -I. if the rules for the default docker0 network exist, iptables -C will behave like the rule already exists even if the interface name is different.
for example, starting with the following filter rules, as reported by iptables -L -n -v, after a docker daemon restart:
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
creating a new network, i'm seeing in the logs (extract):
dockerd[1773]: time="2024-11-28T09:40:37.794204626+02:00" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -o br-f4c3d012b562 -j DOCKER]"
dockerd[1773]: time="2024-11-28T09:40:37.795315461+02:00" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -o br-f4c3d012b562 -j DOCKER]"
but the respective -I command is nowhere to be found. and now the filter rules are:
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
(i.e. unchanged)
for comparison, on startup the equivalent commands for the default network look like:
dockerd[1784]: time="2024-11-27T09:01:43.458188033+02:00" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -o docker0 -j DOCKER]"
dockerd[1784]: time="2024-11-27T09:01:43.459334645+02:00" level=debug msg="/usr/sbin/iptables, [--wait -I FORWARD -o docker0 -j DOCKER]"
even worse, when removing the network, iptables again fails to "match" the output interface (-o br-f4c3d012b562 in this case) and although the command issued is correct, it ends up removing the respective rule for the docker0 network...
dockerd[1773]: time="2024-11-28T09:41:22.439667423+02:00" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -o br-f4c3d012b562 -j DOCKER]"
dockerd[1773]: time="2024-11-28T09:41:22.441201869+02:00" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -o br-f4c3d012b562 -j DOCKER]"
dockerd[1773]: time="2024-11-28T09:41:22.442558725+02:00" level=debug msg="/usr/sbin/iptables, [--wait -D FORWARD -o br-f4c3d012b562 -j DOCKER]"
and the resulting rules are now:
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
(keep in mind, all the logs above are extracts, not the full output, for brevity)
Oh, could be this? ... moby/moby#48844 (comment)
yeap looks like it...
same bug report from the debian tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1087210