Peers disconnect frequently despite running default software

Question

Peers disconnect frequently despite running default software

Opened this issue 18 days ago · 1 comments

There's been an issue for as long as I can remember w/ Comet where peers often disconnect from each other despite all peers operating using unmodified software. For example, in testground we see a large number of peer disconnections despite all nodes in the experiment running stock software. Its worth noting that the default timeouts are being used (45 seconds for receiving a pong)

Here are all of the different errors that resulted in a peer getting disconnected from a single testground experiment

reason send packet msg: write tcp 10.44.240.2:26656->10.33.224.1:53580: write: connection reset by peer reason send packet msg: write tcp 10.44.240.2:26656->10.32.192.3:50992: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.44.240.2:51992->10.36.240.3:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.41.192.2:50130->10.39.0.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.42.176.2:37776->10.32.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.46.0.2:57696->10.34.64.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:50328->10.47.136.3:26656: use of closed network connection reason send packet msg: write tcp 10.46.0.2:57776->10.42.176.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.46.0.2:53824->10.45.72.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:47756->10.43.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:37012->10.39.160.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.0.2:59820->10.34.64.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.46.0.2:26656->10.47.176.3:57254: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.112.3:26656->10.37.208.2:36530: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.36.112.3:53644->10.36.160.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:33182->10.47.232.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.45.72.2:34296: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.112.3:33104->10.45.32.0:26656: use of closed network connection reason send packet msg: write tcp 10.36.112.3:42448->10.43.32.0:26656: use of closed network connection reason send packet msg: write tcp 10.36.112.3:38758->10.34.160.0:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:36818->10.35.96.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:38824->10.46.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.36.240.3:51578: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:50936->10.34.64.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.41.128.2:53860: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:39604->10.32.192.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:36534->10.38.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:51350->10.46.0.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.43.64.2:59764: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.32.0.4:60026: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:26656->10.38.224.1:60170: write: connection reset by peer reason send packet msg: write tcp 10.36.112.3:47088->10.37.112.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.36.112.3:26656->10.43.48.2:48816: use of closed network connection reason switch: reason send packet msg: write tcp 10.36.112.3:26656->10.44.240.2:57366: use of closed network connection reason switch: reason send packet msg: write tcp 10.36.112.3:26656->10.36.160.2:36728: use of closed network connection reason switch: reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.42.16.3:47964: write: connection reset by peer reason send packet msg: write tcp 10.35.224.2:26656->10.38.64.2:33562: write: broken pipe reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.45.16.1:47522: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.41.192.2:37518: write: connection reset by peer reason send packet msg: write tcp 10.35.224.2:26656->10.34.64.1:46558: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.46.64.2:39144: write: connection reset by peer reason recv routine: unexpected EOF reason send packet msg: write tcp 10.35.224.2:59816->10.33.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.224.2:26656->10.42.208.2:58870: write: connection reset by peer reason send packet msg: write tcp 10.35.224.2:26656->10.45.32.0:37068: use of closed network connection reason send packet msg: write tcp 10.35.224.2:34722->10.37.160.1:26656: use of closed network connection reason send packet msg: write tcp 10.35.224.2:26656->10.38.224.1:56240: use of closed network connection reason send packet msg: write tcp 10.35.224.2:54492->10.43.32.0:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.39.0.2:40520: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:38492->10.32.96.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:42964->10.47.176.3:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:35110->10.40.208.2:26656: use of closed network connection reason switch: reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.37.112.2:55134: use of closed network connection reason send packet msg: write tcp 10.35.224.2:42602->10.35.96.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.46.0.2:46380: use of closed network connection reason switch: reason switch: reason send packet msg: write tcp 10.35.224.2:58130->10.44.240.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:42052->10.34.64.1:26656: use of closed network connection reason send packet msg: write tcp 10.35.224.2:26656->10.45.0.2:50006: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.47.192.3:47734: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:37478->10.38.64.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:47272->10.37.32.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.35.224.2:26656->10.34.160.0:43626: use of closed network connection reason switch: reason send packet msg: write tcp 10.33.224.1:52514->10.47.232.2:26656: write: connection reset by peer reason recv routine: unexpected EOF reason send packet msg: write tcp 10.33.224.1:47346->10.38.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.33.224.1:54898->10.39.0.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.33.224.1:26656->10.33.96.2:60568: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.33.224.1:45160->10.32.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: unexpected EOF reason send packet msg: write tcp 10.41.80.3:39498->10.32.224.1:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.41.80.3:26656->10.45.16.1:49578: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.33.96.2:47350->10.46.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.33.96.2:49634->10.41.192.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.33.96.2:26656->10.39.160.1:42156: write: connection reset by peer reason send packet msg: write tcp 10.33.96.2:26656->10.46.0.2:49232: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.208.2:26656->10.43.64.2:55936: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.208.2:41988->10.32.192.3:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.37.208.2:43280->10.42.176.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.37.208.2:26656->10.47.136.3:40000: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:53704->10.39.96.3:26656: write: connection reset by peer reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:26656->10.36.240.3:35330: write: connection reset by peer reason send packet msg: write tcp 10.34.64.1:56166->10.37.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:26656->10.47.136.3:42874: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.34.64.1:26656->10.41.192.2:37630: write: connection reset by peer reason recv routine: unexpected EOF reason send packet msg: write tcp 10.43.48.2:37520->10.43.64.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.43.48.2:26656->10.40.208.2:53286: write: connection reset by peer reason send packet msg: write tcp 10.43.48.2:45092->10.40.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.43.48.2:26656->10.45.80.2:44282: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.96.2:26656->10.35.96.1:49800: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:36382->10.32.0.4:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:26656->10.41.80.3:50112: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:26656->10.33.224.1:33322: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:26656->10.33.96.2:38352: write: broken pipe reason send packet msg: write tcp 10.42.16.3:26656->10.36.240.3:45140: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:35048->10.41.128.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:47920->10.42.176.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:26656->10.45.16.1:56478: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:56504->10.41.128.2:26656: write: broken pipe reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:59102->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:26656->10.37.112.2:36540: write: connection reset by peer reason send packet msg: write tcp 10.42.16.3:56866->10.36.112.3:26656: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.42.16.3:26656->10.45.32.0:58130: use of closed network connection reason send packet msg: write tcp 10.42.16.3:26656->10.38.224.1:34574: use of closed network connection reason send packet msg: write tcp 10.42.16.3:58970->10.46.0.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:36032->10.34.64.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.39.208.2:59066: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:33664->10.35.96.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:47438->10.32.224.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.45.80.2:49130: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:43106->10.37.224.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.45.0.2:55386: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.47.136.3:56628: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:48146->10.46.64.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.45.96.2:53978: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:26656->10.34.160.0:40976: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:37404->10.43.64.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:48292->10.37.208.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.42.16.3:53744->10.44.0.1:26656: use of closed network connection reason switch: reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.32.1:53654->10.45.0.2:26656: write: connection reset by peer reason send routine: pong timeout reason recv routine: unexpected EOF reason send packet msg: write tcp 10.45.16.1:47740->10.40.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.16.1:26656->10.42.176.2:37026: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.32.0.4:26656->10.45.96.2:52746: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.32.0.4:56564->10.44.0.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.47.232.2:36624->10.37.32.1:26656: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.47.232.2:49128->10.40.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.47.232.2:45816->10.36.240.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.208.2:42522->10.35.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.32.96.2:51540->10.46.0.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:50418->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:26656->10.36.160.2:41668: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:26656->10.36.160.2:35294: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:26656->10.46.0.2:52264: write: connection reset by peer reason send packet msg: write tcp 10.32.96.2:57978->10.43.48.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason recv routine: read tcp 10.45.32.0:40298->10.43.32.0:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:38690->10.42.176.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:37068->10.35.224.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:57506->10.43.48.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:36120->10.32.224.1:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:49678->10.37.224.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:58130->10.42.16.3:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.47.232.2:46144: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.32.192.3:48558: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.41.80.3:56200: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.40.224.2:38002: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.37.32.1:49330: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.33.224.1:40688: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.36.112.3:33104: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:43602->10.34.160.0:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:55438->10.38.64.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:55276->10.41.192.2:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.44.0.1:43330: use of closed network connection reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.46.0.2:42710: use of closed network connection reason switch: reason switch: reason recv routine: read tcp 10.45.32.0:26656->10.33.96.2:37696: use of closed network connection reason recv routine: read tcp 10.45.32.0:26656->10.44.240.2:49324: use of closed network connection reason switch: reason send packet msg: write tcp 10.37.224.2:60512->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.37.224.2:55082->10.40.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.37.224.2:52052->10.47.136.3:26656: write: broken pipe reason send packet msg: write tcp 10.41.128.2:26656->10.32.96.2:39690: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.46.64.2:50028->10.47.192.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.46.64.2:26656->10.45.72.2:47956: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.46.64.2:26656->10.36.160.2:45260: write: connection reset by peer reason send packet msg: write tcp 10.46.64.2:26656->10.41.192.2:47322: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.39.160.1:26656->10.32.0.4:50354: write: connection reset by peer reason send packet msg: write tcp 10.45.72.2:53496->10.35.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.72.2:26656->10.34.64.1:36118: use of closed network connection reason send packet msg: write tcp 10.45.72.2:26656->10.40.208.2:60702: write: connection reset by peer reason send packet msg: write tcp 10.45.72.2:58804->10.35.96.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.45.72.2:34476->10.43.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.45.72.2:50080->10.37.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.44.0.1:26656->10.40.224.2:42714: write: connection reset by peer reason send packet msg: write tcp 10.44.0.1:26656->10.36.160.2:49030: write: connection reset by peer reason send packet msg: write tcp 10.44.0.1:26656->10.42.16.3:57810: use of closed network connection reason send packet msg: write tcp 10.44.0.1:26656->10.32.224.1:33820: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.47.192.3:26656->10.46.0.2:56442: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.47.192.3:48878->10.45.16.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.43.64.2:26656->10.34.64.1:48716: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.43.64.2:26656->10.45.96.2:50944: write: broken pipe reason send routine: pong timeout reason send packet msg: write tcp 10.45.0.2:33580->10.47.232.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.45.0.2:60898->10.47.176.3:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.45.0.2:39358->10.35.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.45.0.2:26656->10.33.224.1:51416: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:59852->10.35.224.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:44188->10.41.80.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.36.160.2:57224: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:34992->10.44.0.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.42.176.2:42348: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.45.16.1:44282: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:37428->10.36.112.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:50588->10.39.160.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:41956->10.37.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:59904->10.34.64.1:26656: write: connection reset by peer reason recv routine: unexpected EOF reason send routine: pong timeout reason send packet msg: write tcp 10.39.0.2:26656->10.41.128.2:58036: write: connection reset by peer reason send packet msg: write tcp 10.39.0.2:26656->10.38.224.1:39242: use of closed network connection reason send packet msg: write tcp 10.39.0.2:36122->10.40.224.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:48788->10.37.32.1:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:54566->10.36.160.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:59248->10.40.208.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:58192->10.39.96.3:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:51744->10.34.160.0:26656: use of closed network connection reason switch: reason recv routine: read tcp 10.39.0.2:56630->10.43.32.0:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:40520->10.35.224.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:26656->10.41.192.2:56706: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:47396->10.46.0.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:26656->10.37.160.1:58696: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:26656->10.47.192.3:36646: use of closed network connection reason switch: reason send packet msg: write tcp 10.39.0.2:35992->10.43.48.2:26656: use of closed network connection reason switch: reason send packet msg: write tcp 10.45.80.2:37624->10.47.136.3:26656: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.45.80.2:26656->10.39.0.2:48146: write: connection reset by peer reason send packet msg: write tcp 10.45.80.2:33910->10.37.32.1:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.35.96.1:26656->10.33.96.2:57562: use of closed network connection reason send routine: pong timeout reason send packet msg: write tcp 10.36.240.3:26656->10.47.136.3:43226: write: connection reset by peer reason send packet msg: write tcp 10.36.240.3:38992->10.45.96.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.40.224.2:43406->10.44.0.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.40.224.2:36240->10.39.96.3:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.40.224.2:26656->10.33.96.2:33540: use of closed network connection reason send packet msg: write tcp 10.40.224.2:43308->10.41.80.3:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.40.224.2:26656->10.32.224.1:37638: write: connection reset by peer reason send packet msg: write tcp 10.40.224.2:32790->10.38.64.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.38.64.2:26656->10.36.240.3:42908: write: connection reset by peer reason send packet msg: write tcp 10.38.64.2:26656->10.41.80.3:44398: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:47852->10.42.176.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.160.2:56098->10.47.176.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.160.2:26656->10.45.0.2:51810: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:26656->10.37.32.1:51438: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:40256->10.37.224.2:26656: write: broken pipe reason send packet msg: write tcp 10.36.160.2:40050->10.32.192.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.36.160.2:39410->10.41.192.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.36.160.2:26656->10.45.72.2:37374: write: connection reset by peer reason send routine: pong timeout reason recv routine: unexpected EOF reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.39.96.3:26656->10.45.72.2:48104: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.40.208.2:36954->10.36.160.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.40.208.2:26656->10.42.16.3:42716: write: connection reset by peer reason send packet msg: write tcp 10.40.208.2:43250->10.32.0.4:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:26656->10.36.112.3:34088: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:57556->10.42.16.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:26656->10.43.64.2:34844: write: connection reset by peer reason send packet msg: write tcp 10.42.208.2:57336->10.33.96.2:26656: use of closed network connection reason send packet msg: write tcp 10.42.208.2:26656->10.45.16.1:33824: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.47.136.3:52072->10.47.232.2:26656: use of closed network connection reason send packet msg: write tcp 10.47.136.3:44618->10.33.96.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.47.136.3:44418->10.36.160.2:26656: use of closed network connection reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout reason send packet msg: write tcp 10.32.224.1:34264->10.35.224.2:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:39494->10.47.136.3:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:47612->10.35.96.1:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.32.224.1:38234->10.37.208.2:26656: write: connection reset by peer reason send routine: pong timeout reason send packet msg: write tcp 10.32.224.1:26656->10.36.240.3:38766: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:36570->10.45.72.2:26656: write: broken pipe reason send packet msg: write tcp 10.32.224.1:35292->10.34.64.1:26656: write: connection reset by peer reason send packet msg: write tcp 10.32.224.1:44620->10.44.240.2:26656: write: connection reset by peer reason send routine: pong timeout reason send routine: pong timeout reason send routine: pong timeout

AC

find the cause of most if not all of the above issues and fix them. This would likely involve re-running these traces in knuu considering that is the future of our testing infra, and then addressing them. Considering all nodes in the network tests are behaving honestly and with adequate resources, in theory they should not disconnect.

Answer 1 · 2024-06-10T14:16:44.000Z

It could be the case that nodes are being overwhelmed, in which case debugging bottlenecks in the p2p stack would also address the issues here. Therefore, it makes sense to debug that before moving forward with this.

blocked by #1362