UDP Round Robin Only going to single server
Opened this issue · 7 comments
Dear All,
I am trying to setup UDP round robin load balancing for a SIEM application.
I have seesaw installed and working, I can see packets (using tcpdump on both vm's) going from the vip of 172.16.4.165 to the second server dl-clust-02.
Expected Behaviour
Each udp packet send to the VServer vip (172.16.4.165) from my logging endpoint would be sent to each server in turn i.e dl-clust-01 then dl-clust-02 then back to dl-clust-01 and so on.
Actual Behaviour
UDP packets are only sent to the dl-clust-02.
seesaw.cfg
`[cluster]
anycast_enabled = false
name = defencelogic-lb
node_ipv4 = 172.16.4.163
peer_ipv4 = 172.16.4.164
vip_ipv4 = 172.16.4.160
[config_server]
primary = lb1.
secondary = lb2.
[interface]
node = ens192
lb = ens160`
cluster.pb
`seesaw_vip: <
fqdn: "logger.."
ipv4: "172.16.4.160/24"
status: PRODUCTION
node: <
fqdn: "lb1.."
ipv4: "172.16.4.163/24"
status: PRODUCTION
node: <
fqdn: "lb2.."
ipv4: "172.16.4.164/24"
status: PRODUCTION
vserver: <
name: "logsvr."
entry_address: <
fqdn: "logsvr.."
ipv4: "172.16.4.165/24"
status: PRODUCTION
rp: "ad1@"
vserver_entry: <
protocol: UDP
port: 12201
scheduler: RR
server_low_watermark: 0.3
healthcheck: <
type: ICMP_PING
interval: 5
timeout: 3
retries: 1
>
backend: <
host: <
fqdn: "dl-clust-01.."
ipv4: "172.16.4.61/24"
status: PRODUCTION
>
weight: 1
backend: <
host: <
fqdn: "dl-clust-02.."
ipv4: "172.16.4.62/24"
status: PRODUCTION
>
weight: 1
`
Any helps appreciated.
could you paste what ipvsadm reports?
sudo ipvsadm -Ln
Hi Yuan,
I ran a test using just netcat as shown below to run five requests
echo {"version": "2.1","host":"Request - 4 -test.test.local","short_message":"Helo from seesaw","full_message":"Request 4 here\n\n Load Balancer Testing","level":1,"_user_id":9001,"_user_id":"ped","_some_env_var":"bar"} | ncat -u -w 1 logsvr.<domain fqdn>l 12201
ipvsadm output
admin@lb1:/var/log/seesaw$ sudo ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP 172.16.10.165:12201 rr
-> 172.16.10.61:12201 Route 1 0 3
-> 172.16.10.62:12201 Route 1 0 2
dadmin@lb1:/var/log/seesaw$
I can confirm that test message are routing in a round robin style.
The problem is application related. Our logging agent sends via UDP to the VServer VIP and it seems to lock to a single SIEM back end.
When i send single JSON request as above it works perfectly, the requests are unaltered as I am running tcpdump on backend servers to check for that.
Any ideas why?
It's because connections are tracked and your logging agent is always using the same client side udp port. you can see the connections on the seesaw instance:
sudo ipvsadm -Lnc
Any ideas why?
By default IPVS will track flows and persist each flow (source IP, destination IP, protocol, source port, destination port) to a single backend - often spreading a flow across multiple backends will result in unwanted behaviour/problems (in the case of UDP think video streaming, VoIP calls, etc). As noted, it is likely that your application is using the same source port for all traffic, resulting in what appears to be a single flow (hence being sent to a single backend).
In cases where it is known that the UDP traffic is per packet and that you can safely send each UDP datagram to a different backend (think DNS requests), you can enable "one packet scheduling" (or OPS). This can be enabled on a Seesaw vserver_entry by adding one_packet: true to the configuration.
For our use case, we are using in front of our SIEM solution to provide log balancing for SIEM boxes behind.
As each agent send UDP or log source to the load balancer, the behaviour you described suits my use case. Also with firewallslogs, this is a s a requirement for us as it would mean that we could have large volume of firewall logs sending to a single backend.
If I dont use the one packet option, is there a load balancing protocol to use the least saturated backend server with UDP?
Cheers
I have this working now.
I will post a blog post about how I got it working for others here shortly.
I have enabled one_packet:true in my vserver_entry as per #102 my log sources for my zeek box still will not round robin correctly.
The output of ipvsadm -L -stats shows the following
sudo ipvsadm -L --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
UDP logsvr.test.local:12205 0 116 0 93117 0
-> clust-01.test.local:122 0 50 0 40502 0
-> clust-02.test.local:122 0 66 0 52615 0
UDP 172.16.4.166:12210 0 388 0 209490 0
-> clust-01.test.local:122 0 388 0 209490 0
-> clust-02.test.local:122 0 0 0 0 0
The zeek logs are on port 12210 which are being routed to clust-01.test.local whilst other logs are on 12205 which appear to balance correctly.
Relevant vserver_entry
vserver: <
name: "zeek.test.local"
entry_address: <
fqdn: "logsvr.test.local."
ipv4: "172.16.10.166/24"
status: PRODUCTION
>
rp: "admin@test.local"
vserver_entry: <
protocol: UDP
one_packet: true
port: 12210
scheduler: RR
mode: NAT
healthcheck: <
type: TCP
port: 32999
tls_verify: false
interval: 10
timeout: 5
retries: 3
>
>
backend: <
host: <
fqdn: "clust-01.test.local."
ipv4: "172.16.10.21/24"
status: PRODUCTION
>
weight: 1
>
backend: <
host: <
fqdn: "clust-02.test.local."
ipv4: "172.16.10.22/24"
status: PRODUCTION
>
weight: 1
>
>
I thought that once one_packet: true is enabled that IPVS would not track client source UDP ports.
Any ideas why?