unable to proxy iperf3 traffic

[stevej]

iperf3 is a standard tool used to measure bandwidth performance on hardware switches and routers as well as software TCP proxies.

I'm able to do iperf3 tests between two hosts on the GCE network but not through linkerd-tcp.

Here's my test setup:

iperf3 installed on on netty-test-8, proxy-test-4d, proxy-test-4e.

proxy-test-4e has an iperf server listening on 7474 using the command iperf3 -s -p 7474 -d
proxy-test-4d has a namerd and a linkerd-tcp.
netty-test-8 runs an iperf client: iperf3 -c proxy-test-4d -p 7474 -i 1 -t 30 -b 1M

I'm attempting a 1MB/s test for 30 seconds. This same command works from netty-test-8 to proxy-test-4e` without linkerd-tcp in the mix.

Here is the linkerd-tcp trace log output. My reading of it is that linkerd lb is flapping between 0 connections established and 1 connections established for the iperf server. I'll dig into tcpdump and see what's different about how iperf makes a connection vs how linkerd-tcp makes a connection.

I also have full logfiles available (RUST_LOG=trace) and have attached that
linkerd_tokio_log.txt

[stevej]

I see no connection attempts made from linkerd-tcp

stevej@proxy-test-4e:~$ sudo tcpdump -vvv -A -s 1500 -i ens4 'tcp port 7474' > from_iperf_with_linkerd.txt
tcpdump: listening on ens4, link-type EN10MB (Ethernet), capture size 1500 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel

I do get data with this query from iperf3 directly.

[stevej]

Attached is a pcap of a successful iperf session and an iperf attempt through linkerd
pcaps.zip

As a reminder:
10.240.0.4 is netty-test-8 (iperf client)
10.240.0.21 is proxy-test-4d (linkerd-tcp)
10.240.0.10 is proxy-test-4e (iperf server)

[stevej]

Ok, mystery solved. There were a bunch of rogue slow_cookers running on the cluster against this linkerd-tcp from old load tests and that proxied traffic was leaving the iperf3 server in a bad state unable to respond to traffic.