Will you support nftables?

Question

Will you support nftables?

cybermcm opened this issue 5 years ago · 40 comments

Hi,

Starting with Debian 10 nftables is the current FW solution. I'm managing my FW rules manually (not Docker integrated) and noticed that your fail2ban container uses iptable rules. Is it possible to enhance nftables support?

Answer 1 · 2019-09-29T15:35:21.000Z

Hi @cybermcm, we have to wait fail2ban/fail2ban#2254

Answer 2 · 2019-09-29T20:54:57.000Z

@crazy-max Thanks for your answer, didn't know that. You want me to close this issue or should it stay open in case someone else asks the same question?

Answer 3 · 2019-09-30T03:55:53.000Z

@cybermcm Leave it open for now. I keep you in touch when this feature is implemented.

Answer 4 · 2019-12-18T23:23:21.000Z

It seems the linked PR has been merged now.

Answer 5 · 2019-12-18T23:34:12.000Z

/sbin` # for a in iptables iptables-save  iptables-restore; do ln -sf xtables-nft-multi $a;done

I ran the above inside the container, and this allowed bans to work. My host is Debian 10(buster).

Answer 6 · 2019-12-18T23:49:01.000Z

It seems the linked PR has been merged now.

But not yet released.

Answer 7 · 2020-01-18T17:10:50.000Z

This feature is now available. Let me know if you can test it and give me your feedback.

Answer 8 · 2020-01-19T07:21:04.000Z

@crazy-max Thank you for the information.
I changed my test server to nftables and it seems to work but during my test I noticed that although my own IP gets banned (testing with wrong credentials) I still can access my server. The f2b rule is in place 'table inet f2b-table', all my other rules are stored in 'table ip filter'. I had to change the default table 'inet filter' to 'ip filter' to get docker to continue to set up rules automatically. Maybe this is the reason that banning an IP isn't working?

Answer 9 · 2020-01-19T16:39:47.000Z

@cybermcm I will make some tests too. Keep you in touch.

Answer 10 · 2020-01-19T16:56:22.000Z

Thanks. I tried a few things but didn't get it to work (mainly due to my very basic nftables know-how I think). Question to begin with: With nftables is it still necessary to spin up 2 f2b instances (host and docker)? As far as I understand it shouldn't be necessary any more. Am I right?

Answer 11 · 2020-01-28T10:59:26.000Z

@crazy-max Just wanted to check if you need anything from my side? Can you reproduce my issue?

Answer 12 · 2020-01-28T11:03:01.000Z

I'll tell you when I've got some time

Answer 13 · 2020-02-08T17:59:28.000Z

I spent some time messing with this, and I was able to get this working correctly (as far as I can tell).

I have a jail.d/default.conf file, where I set banaction in the default section:

[DEFAULT]
banaction = nftables[type=multiport]

This is the same for both my fail2ban-input and fail2ban-docker containers (one for host services, the other for docker containers, respectively). At this point, it seems like the fail2ban-input container was working (it bans SSH correctly). fail2ban-docker needs a bit more setup.

In action.d/nftables-common.local (a new file for only the fail2ban-docker container), I have done this:

[Init]
table = f2b-table-docker
chain_hook = forward

table is defined to not overwrite the default f2b-table that the fail2ban-input one generates, chain_hook is forward since I could not get the DOCKER_USER chain to work properly (fail2ban was also deleting the entire chain when it exits).

Now doing nft list ruleset, I get this below all the Docker stuff:

table inet f2b-table {
        set addr-set-sshd {
                type ipv4_addr
                elements = { ... }
        }

        chain f2b-chain {
                type filter hook input priority -1; policy accept;
                tcp dport { ssh } ip saddr @addr-set-sshd drop
        }
}
table inet f2b-table-docker {
        set addr-set-nginx-badbots {
                type ipv4_addr
                elements = { ... }
        }

        chain f2b-chain {
                type filter hook forward priority -1; policy accept;
                tcp dport { http, https } ip saddr @addr-set-nginx-badbots drop
        }
}

Seems to be working!

Answer 14 · 2020-02-08T18:55:02.000Z

@PhasecoreX I tried your suggestion and can confirm that it works. Thanks for this suggestion!
Due to the fact that my knowledge about nft is at a very basic level I can't judge if this is solution has a downside....

Answer 15 · 2020-02-08T19:22:53.000Z

You're welcome! I figure the most basic fail2ban configuration by default should be able to block host stuff correctly (which it does). The downsides for the Docker setup is whatever the downsides of using forward instead of DOCKER-USER is. As I am also new to nft, I am not sure what those downsides are, if at all (as before DOCKER-USER was a thing, we used FORWARD as far as I can remember).

Answer 16 · 2020-06-24T14:00:42.000Z

Hi,
I have a debian running with nft on the host. I have an nginx docker container logging to access.log
I have mounted this file inside the fail2ban docker container.

root@debwaf-nginx2020:~/containers/fail2ban# uname -a
Linux debwaf-nginx2020 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

Fail2ban sees the logging I want to catch (403 errors) and jails the ip-adresses fine:

root@debwaf-nginx2020:~# docker exec -it fail2ban fail2ban-client status nginx403
Status for the jail: nginx403
|- Filter
|  |- Currently failed:	2
|  |- Total failed:	2000
| - File list:	/var/log/nginx/access.log
- Actions
   |- Currently banned:	4
   |- Total banned:	4
   - Banned IP list:	49.235.250.133 178.84.64.116 202.40.191.115 118.24.22.175

However the ip-adresses are not banned in nftables rules:

nft list ruleset
	chain DOCKER-USER {
		counter packets 3436342 bytes 5693372660 return
	}

My config is this:

vi /var/lib/docker/volumes/fail2ban_fail2ban_data/_data/jail.d/jail.local
[INCLUDES]

#before = paths-distro.conf
before = paths-debian.conf

[DEFAULT]
bantime  = 604800 # ban for 7 days

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1/8 192.168.0.0/24 # Your IP address

[nginx403]

enabled = true
port = http,https
filter = nginx403
logpath = /var/log/nginx/access.log
#banaction = docker-action
banaction = nftables[type=multiport]
maxretry = 5
findtime = 120
bantime = 86400

docker-compose is this:

version: '3.4'
services:
  fail2ban:
    image: crazymax/fail2ban:latest
    restart: always
    container_name: fail2ban
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - nginx_log:/var/log/nginx/
      #- fail2ban_conf:/etc/fail2ban/
      - fail2ban_data:/data/
    environment:
      TZ: 'Europe/Amsterdam'
      F2B_DB_PURGE_AGE: 7d



volumes:
  #fail2ban_conf:
  fail2ban_data:
  nginx_log:
     external: true
     name: nginx_nginx_log

logging is a lot like this:

2020-06-24 08:16:31,576 fail2ban.filter         [1]: INFO    [nginx403] Found 118.24.22.175 - 2020-06-24 08:16:31
2020-06-24 08:16:31,822 fail2ban.filter         [1]: INFO    [nginx403] Found 118.24.22.175 - 2020-06-24 08:16:31
2020-06-24 08:16:32,069 fail2ban.filter         [1]: INFO    [nginx403] Found 118.24.22.175 - 2020-06-24 08:16:32
2020-06-24 08:16:32,315 fail2ban.filter         [1]: INFO    [nginx403] Found 118.24.22.175 - 2020-06-24 08:16:32
2020-06-24 08:16:32,830 fail2ban.actions        [1]: WARNING [nginx403] 118.24.22.175 already banned

Howto troubleshoot further? Why doesnt fail2ban block the jailed hosts in nftables on the debian host?

Answer 17 · 2020-06-24T14:31:09.000Z

@hanscees: I'm definitely no expert, I've done some things different but it is working:

compose file:
missing: network_mode: "host"
missing env: - F2B_IPTABLES_CHAIN=DOCKER-USER (not really sure if this is necessary)
create action.d/nftables-common.local file with

[Init]
table = f2b-table-docker
chain_hook = forward

see comment #29 (comment)

blocking IPs are added to table inet f2b-table in this case

Answer 18 · 2020-06-26T10:56:34.000Z

I do have the nftables, now I set networking to host.

However nothing is blocked. This probably has to do with my nftables setup using table inet.
have a look here:
moby/moby#26824

this docker-compose.yml works:

version: '3.4'
services:
  fail2ban:
    image: crazymax/fail2ban:latest
   #restart: always
    container_name: fail2ban
    network_mode: host
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - nginx_log:/var/log/nginx/
      #- fail2ban_conf:/etc/fail2ban/
      - fail2ban_data:/data/
    environment:
      TZ: 'Europe/Amsterdam'
      F2B_DB_PURGE_AGE: 7d



volumes:
  #fail2ban_conf:
  fail2ban_data:
  nginx_log:
     external: true
     name: nginx_nginx_log

Answer 19 · 2020-06-26T21:47:32.000Z

I do have the nftables, now I set networking to host.

However nothing is blocked. This probably has to do with my nftables setup using table inet.
have a look here:
moby/moby#26824

this docker-compose.yml works:

version: '3.4'
services:
  fail2ban:
    image: crazymax/fail2ban:latest
   #restart: always
    container_name: fail2ban
    network_mode: host
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - nginx_log:/var/log/nginx/
      #- fail2ban_conf:/etc/fail2ban/
      - fail2ban_data:/data/
    environment:
      TZ: 'Europe/Amsterdam'
      F2B_DB_PURGE_AGE: 7d



volumes:
  #fail2ban_conf:
  fail2ban_data:
  nginx_log:
     external: true
     name: nginx_nginx_log

My conclusion "nothing is blocked" was wrong.

After adding host networking:

I did see a jail by checking via docker exec -it fail2ban fail2ban-client status
the jail could be checked and held ip-addresses docker exec -it fail2ban fail2ban-client status nginx403
On the host the nftables table was there: nft list table inet f2b-table-docker -n

table inet f2b-table-docker { # handle 21
	set addr-set-nginx403 { # handle 2
		type ipv4_addr
		elements = { 129.211.22.74, 213.41.135.119 }
	}

	chain f2b-chain { # handle 1
		type filter hook forward priority -1; policy accept;
		ip saddr @addr-set-nginx403 ip protocol tcp counter packets 0 bytes 0 tcp dport { http, https } drop # handle 9
		tcp dport { http, https } ip saddr @addr-set-nginx403 reject # handle 6
		tcp dport ssh counter packets 0 bytes 0 # handle 7
		tcp dport { http, https } counter packets 1 bytes 52 # handle 11
	}
}

notice I added the lines with handle 9, 7 and 11 manually by doing

nft add rule inet f2b-table-docker f2b-chain tcp dport { http, https } counter
nft insert rule inet f2b-table-docker f2b-chain position 6 ip saddr @addr-set-nginx403 ip protocol tcp counter tcp dport { http, https } drop

My issue was that maxretry was on 5 instead of 1 (I expected an immediate ban). After I set maxretry to 1 the bans are immediate as I want them to be.

Thanks for the help! nftables tested ok here!

Answer 20 · 2020-08-05T04:05:02.000Z

Seeing as #46 and #17 were closed recently, I decided to revisit this to get nftables working with just one container for both input and forward chains. I think I got it working, and I am really happy with the results. With this setup, Fail2Ban makes one table, but makes 2 chains in it, one for input (host), and one for forward (docker).

First, make 2 files in the action.d folder:

action.d/nftables-input.conf

[INCLUDES]
before = nftables.conf

[Init]
chain = f2b-chain-input
chain_hook = input

action.d/nftables-forward.conf

[INCLUDES]
before = nftables.conf

[Init]
chain = f2b-chain-forward
chain_hook = forward

And that's it. For each of your jails, instead of using chain = INPUT or chain = DOCKER-USER, you would use banaction = nftables-input or banaction = nftables-forward, depending on if you are protecting a host service or a docker service respectively. There is no need to specify chain = ??? in the jail, as the banaction will take care of that. Here are the two examples given (sshd and traefik) but for nftables instead:

[sshd]
enabled = true
banaction = nftables-input
port = ssh
filter = sshd[mode=aggressive]
logpath = /var/log/auth.log
maxretry = 5

[traefik-auth]
enabled = true
banaction = nftables-forward
port = http,https
filter = traefik-auth
logpath = /var/log/traefik/access.log

[traefik-botsearch]
enabled = true
banaction = nftables-forward
port = http,https
filter = traefik-botsearch
logpath = /var/log/traefik/access.log

Hope this helps!

Answer 21 · 2020-08-05T07:38:58.000Z

@PhasecoreX That's really nice, thanks for that! I think we can move forward and implement this behavior in this image.

Answer 22 · 2020-08-29T14:43:41.000Z

Hi guyz,

It works for me too with nftables-multiport.conf instead of nftables.conf

Thank you for your work !

Answer 23 · 2020-09-29T18:00:28.000Z

Small feedback coming from a different configuration from those already mentioned here.

I am under Fedora 32 Server with nftables and I am using Podman as well as SeLinux in Enforcing mode.

The container is launched via a systemd service (if some people are interested, I could add my unit).
Fail2ban is configured taking into account the information provided by @PhasecoreX (thanks to him)
And for the SeLinux rules, I generated them via the Udica utility.

For now, I have not yet set up all my jails but the one for sshd works perfectly and the IPs are correctly banned.

Answer 24 · 2021-04-03T04:21:32.000Z

Hoping someone can help me with my Debian 10 nftables setup. I'm trying to do it the way @PhasecoreX described.

Relevant compose:

  fail2ban:
    container_name: fail2ban
    hostname: fail2ban
    image: crazymax/fail2ban:latest
    network_mode: "host"
    cap_add:
      - "NET_ADMIN"
      - "NET_RAW"
    volumes:
      - "/docker/config/fail2ban:/data"
      - "/etc/localtime:/etc/localtime:ro"
      - "/var/log/auth.log:/var/log/auth.log:ro"
    environment:
      - "TZ=America/New_York"
    restart: unless-stopped
    logging:
      driver: json-file
      options: 
        max-file: "3"
        max-size: "10M"

My action.d/nftables-forward.conf:

[INCLUDES]
before = nftables.conf

[Init]
chain = f2b-chain-forward
chain_hook = forward

My action.d/nftables-input.conf:

[INCLUDES]
before = nftables.conf

[Init]
chain = f2b-chain-input
chain_hook = input

My jail.d/sshd.conf:

[sshd]
enabled = true
banaction = nftables-input
port = ssh
filter = sshd[mode=aggressive]
logpath = /var/log/auth.log
maxretry = 5

There are banned IP's by my jail:

root@Vergil:~# docker exec -it fail2ban fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  |- Total failed:     38
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 2
   |- Total banned:     6
   `- Banned IP list:   36.80.211.9 180.246.127.205
root@Vergil:~#

But nothing in my iptables:

root@Vergil:~# iptables -S | grep f2b
# Warning: iptables-legacy tables present, use iptables-legacy to see them
root@Vergil:~# iptables-legacy -S | grep f2b
root@Vergil:~#

Any idea where I've gone wrong?

Answer 25 · 2021-04-03T06:28:06.000Z

@goose-ws: your config seems fine. To view your bans try nftable tools, like
nft list ruleset
zero results with iptables command are perfectly fine if you switched to nftables

Answer 26 · 2021-04-03T11:29:55.000Z

It appears that I don't have nft installed, despite being on a recent install of Debian 10

root@Vergil:~# nft list ruleset
bash: nft: command not found
root@Vergil:~# which nft
root@Vergil:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@Vergil:~# uname -r
4.19.0-14-amd64
root@Vergil:~#

Answer 27 · 2021-04-03T11:39:44.000Z

Did you update an old installation because then iptables stays.
If you want to switch then take a look at Debian nftables, but take care that you can reach your machine locally if you play around with your rules. I locked myself out about a hundred times ;-)

Answer 28 · 2021-10-03T10:21:03.000Z

I followed the config from @PhasecoreX but can't get nftables blocking anything. Testing off my mobile connection but still getting able to get load services served by Traefik:

$ docker logs -f fail2ban

...

2021-10-03 21:13:13,041 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:13
2021-10-03 21:13:14,144 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:14
2021-10-03 21:13:14,286 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:14
2021-10-03 21:13:14,673 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:14
2021-10-03 21:13:15,704 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:15
2021-10-03 21:13:15,829 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:15
2021-10-03 21:13:15,951 fail2ban.actions        [1]: NOTICE  [traefik-auth] Ban 120.18.36.14
2021-10-03 21:13:17,434 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:17
2021-10-03 21:13:18,510 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:18
2021-10-03 21:13:18,650 fail2ban.filter         [1]: INFO    [traefik-auth] Found 120.18.36.14 - 2021-10-03 21:13:18

$ sudo nft list ruleset

...

table inet f2b-table {
        set addr-set-traefik-auth {
                type ipv4_addr
                elements = { 120.18.36.14 }
        }

        chain f2b-chain-forward {
                type filter hook forward priority filter - 1; policy accept;
                tcp dport { 80, 443 } ip saddr @addr-set-traefik-auth reject
        }
}

Answer 29 · 2021-10-03T19:19:13.000Z

@calvinbui: Seems correct for me (but I'm no expert at all). Maybe another rule in place which interferes with the F2B rule?

Answer 30 · 2021-10-04T02:48:11.000Z

I can get sshd being blocked successfully (INPUT chain) but not anything through to Docker.

I tried using the legacy method with update-alternatives but had the same issue.

Reading the docs, it mentions the FORWARD is evaluated after the DOCKER and DOCKER-USER chains. That would be the main difference between the 'legacy' implementation and this new way. Those chains were empty for me anyway:

table ip filter {
        ...
        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                counter packets 5891 bytes 8245496 jump DOCKER-USER
                counter packets 5891 bytes 8245496 jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter packets 3262 bytes 8105011 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 2629 bytes 140485 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
                counter packets 0 bytes 0 jump LIBVIRT_FWX
                counter packets 0 bytes 0 jump LIBVIRT_FWI
                counter packets 0 bytes 0 jump LIBVIRT_FWO
        }
        ...
        chain DOCKER {
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 2629 bytes 140485 jump DOCKER-ISOLATION-STAGE-2                      counter packets 5891 bytes 8245496 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "docker0" counter packets 0 bytes 0 drop
                counter packets 2629 bytes 140485 return
        }

        chain DOCKER-USER {
                counter packets 5891 bytes 8245496 return
        }
}

table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 138 bytes 8581 jump DOCKER
        }

        ...
        chain OUTPUT {
                type nat hook output priority -100; policy accept;
                ip daddr != 127.0.0.0/8 fib daddr type local counter packets 0 bytes 0 jump DOCKER
        ...
        chain DOCKER {
                iifname "docker0" counter packets 0 bytes 0 return
        }
}

Answer 31 · 2021-10-04T04:35:10.000Z

figured it out, iptables doesn't work on containers using macvlan.

Answer 32 · 2021-12-10T12:11:04.000Z

I can confirm that. I also have MacVlan installed. I can see in the log that the IP was recognized and should be blocked. However, there is no action. This means that the page can still be accessed for the blocked IP. Does anyone know a solution?

Answer 33 · 2022-02-10T21:30:29.000Z

Same problem here! I have an nginx reveres proxy container with a macvlan network and a docker bridge network before a vaultwarden with only the same bridge network. I got the events in the fail2ban log and also the ban, but I can still access the vaultwarden through the proxy. Any way to fix that?

                       --> Host  Network                             --> Fail2Ban Container
                     /
                   eth0
                   /
Router --> RP4 --> eth0.80 --> MACVLAN eth0.80 --> Ngynx Proxy Container --> Docker Bridge --> Vaultwarden Container

Running on RP4+ 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

Answer 34 · 2023-01-11T09:10:07.000Z

I have the same issue with MACVLAN... Did anyone of you ever fix this?
@calvinbui @Cavekeeper @TheUntouchable ?

Answer 35 · 2023-01-11T09:15:59.000Z

Unfortunately no, but I'm switching from npm to traefik now and will block access via CrowdSec for additional security without iptables or nftables.
Have a look here if this is interesting for you:
https://www.youtube.com/watch?v=-GxUP6bNxF0&list=FLwYuJ29dMnJ0kIsYu1tskFA

Answer 36 · 2023-01-13T04:42:03.000Z

I have the same issue with MACVLAN... Did anyone of you ever fix this? @calvinbui @Cavekeeper @TheUntouchable ?

I updated alternatives to iptables-legacy to get it working

Answer 37 · 2023-01-19T12:59:02.000Z

I have the same issue with MACVLAN... Did anyone of you ever fix this? @calvinbui @Cavekeeper @TheUntouchable ?

I updated alternatives to iptables-legacy to get it working

Hmm, interesting...
For me it shows that it is doing the ban, it creates the rule properly, but I can still access the container then, which is weird.

Answer 38 · 2023-01-19T13:30:57.000Z

I don't think @calvinbui is using macvlans, as here the local firewall will not kick in anyway

Answer 39 · 2023-01-19T22:37:07.000Z

I don't think @calvinbui is using macvlans, as here the local firewall will not kick in anyway

I am using macvlan for all my containers, but the fail2ban container is running in host mode.

https://github.com/calvinbui/ansible-monorepo/blob/master/fail2ban.yml

Answer 40 · 2023-01-20T07:34:59.000Z

Mh okay. I had fail2ban also in host mode and also tried to switch the alternatives to iptable-legacy, but I am using a raspberrypi 4 with a Raspberry 64Bit OS and somehow there is a problem with ip/nftables on that I have the feeling.. :(