MaxKellermann/ferm

Keep rule on flush

beppler opened this issue · 7 comments

beppler commented

Is it possible to block the remove of a rule during flush?

I'm using ferm to configure the firewal on a machine running docker.

In this configuration there is a filter chain named DOCKER-USER that is managed my the user, but it is required by docker.

it is configured by docker to be domain (ip ip6) table filter chain DOCKER-USER { jump RETURN; }.

I reconfigured it using ferm, but when I run it with --flush the DOCKER-USER chain is deleted and the command fail because of a jump to this chain in the docker iptables configuration.

MaxKellermann commented

Use @preserve (the manual has a section about it).

beppler commented

Ok, I tried to use preserve, but the rules are created after that jump RETURN created by docker.

This is the result of iptables -S just after docker is started.

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

And my configuration file.

@def $EXT_DEV = ens256;
@def $BKP_DEV = ens224;

domain (ip ip6) {
    table nat {
        # preserve docker rules
        chain (DOCKER DOCKER-INGRESS PREROUTING OUTPUT POSTROUTING) @preserve;
    }

    table filter {
        # preserve docker rules
        chain (DOCKER DOCKER-INGRESS DOCKER-ISOLATION-STAGE-1 DOCKER-ISOLATION-STAGE-2 FORWARD) @preserve;

        chain INPUT {
            policy DROP;

            # connection tracking
            mod state state INVALID DROP;
            mod state state (ESTABLISHED RELATED) ACCEPT;

            # allow local packet
            interface lo ACCEPT;

            # respond to ping
            proto icmp ACCEPT;

            interface $EXT_DEV {
                # allow SSH connections
                saddr 172.30.0.0/17 proto tcp dport ssh ACCEPT;

                # allow SMTP connections
                proto tcp dport (smtp submission) ACCEPT;

                # allow HTTP/HTTPS connections
                proto tcp dport (http https) ACCEPT;
            }

            # allow Data Protector connections
            interface $BKP_DEV saddr 192.168.172.100 proto tcp dport (ssh omni) ACCEPT;
        }

        chain OUTPUT {
            policy ACCEPT;

            # connection tracking
            mod state state INVALID DROP;
            mod state state (ESTABLISHED RELATED) ACCEPT;
        }

        # docker user container rules
        chain DOCKER-USER @preserve;
        chain DOCKER-USER {
            # connection tracking
            mod state state INVALID DROP;
            mod state state (ESTABLISHED RELATED) ACCEPT;

            # allow HTTP/HTTPS connections
            proto tcp dport (http https) ACCEPT;
        }
    }
}

The result iptables -S is:

-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A INPUT -m state --state INVALID -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -s 172.30.0.0/17 -i ens256 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i ens256 -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -i ens256 -p tcp -m tcp --dport 587 -j ACCEPT
-A INPUT -i ens256 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i ens256 -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -s 192.168.172.100/32 -i ens224 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.172.100/32 -i ens224 -p tcp -m tcp --dport 5555 -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A DOCKER-USER -m state --state INVALID -j DROP
-A DOCKER-USER -m state --state RELATED,ESTABLISHED -j ACCEPT
-A DOCKER-USER -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-USER -p tcp -m tcp --dport 443 -j ACCEPT

Not that ferm rules are created after the original -A DOCKER-USER -j RETURN.

And when I reload ferm rules all the configuration made on DOCKER-USER is duplicated.

That is why I stopped to use @preserve.

But when I stopped it there where an error when I try to flush the rules because of -A FORWARD -j DOCKER-USER rule created by docker.

beppler commented

I made it work adding the following hooks in addition to @preserve.

@hook pre "iptables -F DOCKER-USER; ip6tables -F DOCKER-USER";
@hook post "iptables -D DOCKER-USER -j RETURN; ip6tables -D DOCKER-USER -j RETURN";
@hook flush "iptables -F DOCKER-USER; ip6tables -F DOCKER-USER; iptables -A DOCKER-USER -j RETURN; ip6tables -A DOCKER-USER -j RETURN";
nlvw commented

@beppler any chance you could post your complete config? I'm struggling with a similar issue.

beppler commented

Here is it (with some comments

@def $EXT_DEV = ens256;
@def $BKP_DEV = ens224;

domain (ip ip6) {
    table nat {
        # preserve docker rules
        chain (DOCKER DOCKER-INGRESS PREROUTING OUTPUT POSTROUTING) @preserve;
    }

    table filter {
        # preserve docker rules
        chain (DOCKER DOCKER-INGRESS DOCKER-ISOLATION-STAGE-1 DOCKER-ISOLATION-STAGE-2 FORWARD) @preserve;

        chain INPUT {
            policy DROP;

            # connection tracking
            mod state state INVALID DROP;
            mod state state (ESTABLISHED RELATED) ACCEPT;

            # allow local packet
            interface lo ACCEPT;

            # respond to ping
            proto icmp ACCEPT;

            interface $EXT_DEV {
                # allow SSH connections
                saddr 172.30.0.0/17 proto tcp dport ssh ACCEPT;

                # allow HTTP/HTTPS connections
                proto tcp dport (http https) ACCEPT;
            }

            # allow SMTP connections (from docker0 ip range)
            saddr 172.16.0.0/12 proto tcp dport (smtp submission) ACCEPT;

            # allow Data Protector connections
            interface $BKP_DEV saddr 192.168.172.100 proto tcp dport (ssh omni) ACCEPT;
        }

        chain OUTPUT {
            policy ACCEPT;

            # connection tracking
            mod state state INVALID DROP;
            mod state state (ESTABLISHED RELATED) ACCEPT;
        }

        # docker user container rules
        # use @preserve to avoid error on flush
        chain DOCKER-USER @preserve; 
        chain DOCKER-USER {
            # connection tracking
            mod state state INVALID DROP;
            mod state state (ESTABLISHED RELATED) ACCEPT;

            # block external -> container access
            DROP;
        }
    }

    # clear DOCKER-USER to avoid duplicated rules because of @preserve
    @hook pre "iptables -F DOCKER-USER; ip6tables -F DOCKER-USER";
    # remove the original return created by docker (preserved by @preserve)
    @hook post "iptables -D DOCKER-USER -j RETURN; ip6tables -D DOCKER-USER -j RETURN";
    # restore original DOCKER-USER rules after flush 
    @hook flush "iptables -F DOCKER-USER; ip6tables -F DOCKER-USER; iptables -A DOCKER-USER -j RETURN; ip6tables -A DOCKER-USER -j RETURN";
}
  • 👍1
powerman commented

I've managed to configure ferm to keep docker rules even in case we need to have custom rules in other docker-modified chain than filter/FORWARD (in which we already have DOCKER-USER chain out of box).

Here is example for having own rules both in filter/FORWARD and nat/POSTROUTING.

# Docker support.
# To remove docker support later:
# - remove rules under this comment
# - move rules from filter/DOCKER-USER to filter/FORWARD, if any
# - move rules from all other {table}/{chain}-DOCKER-USER to {table}/{chain}, if any
table nat chain (PREROUTING OUTPUT POSTROUTING DOCKER) @preserve;
table filter chain (FORWARD DOCKER DOCKER-ISOLATION-STAGE-1 DOCKER-ISOLATION-STAGE-2) @preserve;
table nat chain (PREROUTING-DOCKER-USER OUTPUT-DOCKER-USER POSTROUTING-DOCKER-USER) {}
@hook post "iptables -t nat -C PREROUTING  -j PREROUTING-DOCKER-USER  2>/dev/null || iptables -t nat -A PREROUTING  -j PREROUTING-DOCKER-USER";
@hook post "iptables -t nat -C OUTPUT      -j OUTPUT-DOCKER-USER      2>/dev/null || iptables -t nat -A OUTPUT      -j OUTPUT-DOCKER-USER";
@hook post "iptables -t nat -C POSTROUTING -j POSTROUTING-DOCKER-USER 2>/dev/null || iptables -t nat -A POSTROUTING -j POSTROUTING-DOCKER-USER";

table nat chain POSTROUTING-DOCKER-USER {
    # our custom rules for nat/POSTROUTING goes here, e.g.:
    outerface eth0 MASQUERADE;
}
table filter chain DOCKER-USER {
    # our custom rules for filter/FORWARD goes here, e.g.:
    protocol tcp dport 25 REJECT reject-with icmp-port-unreachable;
}

@MaxKellermann It may worth to add this example to ferm docs, because using it together with docker is a common enough case (docker is everywhere nowadays) and also is a pain.

Related moby/moby#40544

  • 🎉1
Deleted user commented

@beppler thanks for your config, it helped me quite a lot

My only issue left is when reloading ferm with systemctl reload ferm, my rules in the DOCKER-USER chain are duplicated. When performing a restart, the duplicates are removed correctly.

Since you mentioned this behavior in a previous comment regarding the use of @preserve I'm wondering if there's an error in my config or the issue still persists.