Nordix/nfqueue-loadbalancer

A load-balancer based on the NFQUEUE iptables target. The -j NFQUEUE target directs packets to a user-space program. The program can analyze the packet and set fwmark and verdict.

The nfqlb lb program receives packets and uses a configuration in shared memory to compute a fwmark. The nfqlb program invoked with other commands configures the shared memory. This decouples the traffic handling from configuration.

The fwmark represents a real-target and forwarding of packets is done by Linux routing or NAT based on the fwmark. A basic setup example;

# initiate the shared mem structure
nfqlb init
# Redirect packets to the VIP address to the nfqueue
iptables -t mangle -A PREROUTING -d 10.0.0.1/32 -j NFQUEUE --queue-num 2

# Add NAT/routing and activate fwmark (repeat for all real-targets)
if test "$DSR" = "yes"; then
  # Use routing for Direct Server Return (DSR)
  ip rule add fwmark 1 table 1
  ip route add default via 192.168.1.1 table 1
else
  # NAT based load-balancing
  iptables -t nat -A POSTROUTING -m mark --mark 1 -j DNAT --to-destination 192.168.1.1
fi
nfqlb activate 1

# Check the config and start load-balancing
nfqlb show
nfqlb lb

When a real-target is lost it must be removed from the configuration with;

nfqlb deactivate 5

Automatic detection and re-configuration when a target (or load-balancer) is lost/added is not a part of nfqueue-loadbalancer. You must do that in your own way.

Hashing and fragment handling is done in the same way as the Google load-balancer; Maglev. nfqlb is stateless and since the configuration is the same for all instances it does not matter where packets are received. This makes nfqlb scalable.

More info

Maglev - How functions from the Google load-balancer is used
Fragment handling - How fragmented packets are handled
Improved TCP performance - For TCP only SYN packets may be load-balanced
Fragment tracking - With another hash table
Testing - Unit, function and performance testing
Destination Unreachable icmp - The PMTU discovery problem
SCTP - SCTP load-balancing with multihoming
extend/alter - Extend or alter nfqlb
Flows - Define LBs for flows, (proto,src,dst,sport,dport) tuples
Log/trace - Log and trace

Try it

Try it on your own machine using Docker containers.

You must have some targets. We use docker containers, but anything with an ip-address (not loopback) will do.

Bring up some container targets;

for n in 1 2 3; do
  name=target$n
  docker run -d --hostname=$name --name=$name --rm alpine:latest nc -nlk -p 8888 -e hostname
  docker inspect $name | jq .[].NetworkSettings.Networks.bridge.IPAddress
  #docker inspect $name | jq .[].NetworkSettings.Networks.bridge.GlobalIPv6Address
done
nc <addr> 8888   # Test connectivity

You can setup load-balancing in your main network name-space but it is better to use another container when testing. Start the test container and enter. You must use --privileged for network configuration.

docker run --privileged -it --rm registry.nordix.org/cloud-native/nfqlb:latest /bin/sh

Use the nfqlb.sh lb script to setup load-balancing. The setup is NAT based like the example above, check the cmd_lb() function in nfqlb.sh for details.

In the test container;

PATH=$PATH:/opt/nfqlb/bin
nfqlb.sh lb --vip=10.0.0.0/32 <your container ip addresses here...>
# (example; nfqlb.sh lb --vip=10.0.0.0/32 172.17.0.3 172.17.0.4 172.17.0.5)

# Check load-balancing;
for n in $(seq 1 20); do echo | nc 10.0.0.0 8888; done

# Check some things
iptables -t nat -S    # OUTPUT chain for local origin, forwarding is not setup!
iptables -t mangle -S # The VIP is routed to user-space
nfqlb show            # Shows the Maglev hash lookup
nfqlb deactivate 101  # Deactivates a target. Check load-balancing again!

Stop the targets;

docker stop target1 target2 target3

This is a basic example. nfqlb can load-balance forwarded traffic also (of course) and do Direct Server Return (DSR). This will however require a different setup. But the setup in this example can be used for IPv6. You must setup ipv6 support in docker and use ipv6 addresses for the VIP and targets in the nfqlb.sh lb command. But that is left as an exercise for the reader.

You can also test with xcluster using the function tests.

Build

make -C src help
make -C src -j8
./nfqlb.sh build_image    # Build the test docker image

Linked with -lmnl -lnetfilter_queue so you must install those. Libpcap is used in unit-tests.

sudo apt install -y libmnl-dev libnetfilter-queue-dev libpcap-dev

Static binary;

# On Ubuntu 20.04
./nfqlb.sh libnfqueue_download
./nfqlb.sh libnfqueue_unpack
./nfqlb.sh libnfqueue_build
# On Ubuntu 22.04
./nfqlb.sh libmnl_download
./nfqlb.sh libmnl_unpack
./nfqlb.sh libmnl_build

make -C src clean
make -C src -j8 static
strip /tmp/$USER/nfqlb/nfqlb/nfqlb
file /tmp/$USER/nfqlb/nfqlb/nfqlb
./nfqlb.sh build_alpine_image   # Will build a static binary

The static libs are not present for libnfqueue-queue-dev in Ubuntu 20.04. However they are present in Ubuntu 22.04, but then the static libs are not in libmnl-dev (sigh...). This bug is reported.