swift-server/async-http-client

AHC might have a problem with remotes that close the connection

Closed this issue · 1 comments

t089 commented

I have a web service that uses AHC to download image files from a remote server. In most of the cases this remote server is a single AWS CloudFront distribution. The service runs well with an average throughput of a few req/s and peak at around 50 req/s. There are however quite a few errors reported from AHC: IOError: read(descriptor:pointer:size:): Connection reset by peer (errno: 104). They seem to appear mostly at times of low throughput. After chatting with @fabianfett we suspected that the connection pool might not close and drop connections, if the remote sends a connection: close header. Anyhow, this issue is more of a reminder to check the behaviour after the currently ongoing refactor of the connection pool and to make sure it is covered by a test case.

FWIW: This week this error has not yet happened again in my service. So maybe, this was also some random faulty behaviour on CloudFront?! The errors still show up occasionally.

For reference, a similar issue was reported on the aws go sdk: aws/aws-sdk-go#2825

fabianfett commented

Anyhow, this issue is more of a reminder to check the behaviour after the currently ongoing refactor of the connection pool and to make sure it is covered by a test case.

Are we there yet? Tests that should cover this scenario now:

  • HTTP1ConnectionStateMachineTests.testAConnectionCloseHeaderInTheRequestLeadsToConnectionCloseAfterRequest
  • HTTP1ConnectionStateMachineTests.testAConnectionCloseHeaderInTheResponseLeadsToConnectionCloseAfterRequest
  • HTTP1ConnectionTests.testConnectionClosesOnRandomlyAppearingCloseHeader
  • HTTP1ConnectionTests.testConnectionClosesOnCloseHeader
  • HTTPClientTests.testRemoteClose
  • HTTPClientTests.testSubsequentRequestsWorkWithServerSendingConnectionClose
  • HTTPClientTests.testSubsequentRequestsWorkWithServerAlternatingBetweenKeepAliveAndClose
  • HTTPClientTests.testRepeatedRequestsWorkWhenServerAlwaysCloses
  • HTTPClientTests.testWeHandleUsSendingACloseHeaderCorrectly
  • HTTPClientTests.testWeHandleUsReceivingACloseHeaderAmongstOtherConnectionHeadersCorrectly

We might need to reopen this once we have more information.