Periodic waves of EOF every 60 seconds.
stevej opened this issue · 3 comments
stevej commented
testing linkerd-tcp with the stock example.yml and slow_cooker at 20k qps is resulting in large waves of EOFs.
Get http://proxy-test-4d:7474: EOF [96/25466]
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:53:27Z 19829/0/181 20000 99% 10s 0 [ 2 4 5 18 ] 18 0
...
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: read tcp 10.240.0.4:47703->10.240.0.21:7474: read: connection reset by peer
2017-06-28T18:54:27Z 19819/0/181 20000 99% 10s 0 [ 2 4 6 14 ] 14 0
...
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:55:27Z 19817/0/182 20000 99% 10s 0 [ 1 4 5 206 ] 206 0
...
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:56:27Z 19828/0/172 20000 99% 10s 0 [ 2 4 6 9 ] 9 0
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
2017-06-28T18:57:27Z 19839/0/161 20000 99% 10s 0 [ 3 5 7 9 ] 9 0
Interestingly, they seem to come in groups of 170-200 at a time about every 60 seconds.
stevej commented
High concurrency even at low qps seems to cause this
stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -qps 1 -concurrency 1000 http://proxy-test-4d:7474
# sending 1000 GET req/s with concurrency=1000 to http://proxy-test-4d:7474 ...
# good/b/f t goal% min [p50 p95 p99 p999] max bhash change
2017-06-29T20:23:24Z 9000/0/0 10000 90% 10s 0 [ 17 105 257 291 ] 291 0 +
2017-06-29T20:23:34Z 10000/0/0 10000 100% 10s 0 [ 7 15 16 22 ] 22 0
2017-06-29T20:23:44Z 10051/0/0 10000 100% 10s 0 [ 18 26 26 32 ] 32 0
2017-06-29T20:23:54Z 10272/0/0 10000 100% 10s 1 [ 11 25 26 30 ] 30 0
2017-06-29T20:24:04Z 9772/0/0 10000 97% 10s 0 [ 14 25 26 29 ] 29 0
2017-06-29T20:24:14Z 10004/0/0 10000 100% 10s 1 [ 16 25 26 30 ] 30 0
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
Get http://proxy-test-4d:7474: EOF
...
2017-06-29T20:24:24Z 9729/0/561 10000 97% 10s 1 [ 12 43 209 220 ] 220 0
again at the 1 minute mark.
wmorgan commented
egg timer bug
stevej commented
I haven't dug into tcpdump yet but I'll note this doesn't happen when the backends and test client are all on localhost.