swift-server/async-http-client

StreamClosed location NIOHTTP2/HTTP2StreamChannel.swift:827

jake-betplusev opened this issue · 6 comments

jake-betplusev commented

Hi all!

I've been having an issue recently with my Vapor ScheduledJobs failing for the following reasons (not always the same reason and this isn't an exhaustive list afaik):

  • HTTPClientError.cancelled
  • HTTPClientError.remoteConnectionClosed
  • StreamClosed(streamID: HTTP2StreamID(203), errorCode: HTTP2ErrorCode<0x8 Cancel>, location: "NIOHTTP2/HTTP2StreamChannel.swift:827")
  • StreamClosed(streamID: HTTP2StreamID(205), errorCode: HTTP2ErrorCode<0x8 Cancel>, location: "NIOHTTP2/HTTP2StreamChannel.swift:827")
  • StreamClosed(streamID: HTTP2StreamID(201), errorCode: HTTP2ErrorCode<0x8 Cancel>, location: "NIOHTTP2/HTTP2StreamChannel.swift:827")

I came across the following similar reported issues:

After reading those issues, I tried setting app.http.client.configuration.httpVersion = .http1Only in configure.swift and that appears to have fixed my issue 🤞.

I'm using Vapor version 4.0.0 and async-http-client version 1.12.0. I'm still using EventLoopFuture and have not yet migrated to async / await.

Please let me know if there is any information I can provide to help resolve this issue so I can remove the reliance on http1Only. Thank you!

dnadoba commented

So HTTPClientError.cancelled usually indicates that we likely hit some timeout or other reason vapor or user code has canceled the request.

The other errors:

  • HTTPClientError.remoteConnectionClosed
  • StreamClosed(streamID: HTTP2StreamID(203), errorCode: HTTP2ErrorCode<0x8 Cancel>, location: "NIOHTTP2/HTTP2StreamChannel.swift:827")
  • StreamClosed(streamID: HTTP2StreamID(205), errorCode: HTTP2ErrorCode<0x8 Cancel>, location: "NIOHTTP2/HTTP2StreamChannel.swift:827")
  • StreamClosed(streamID: HTTP2StreamID(201), errorCode: HTTP2ErrorCode<0x8 Cancel>, location: "NIOHTTP2/HTTP2StreamChannel.swift:827")

indicate that the server has closed the connection.

The StreamIDs are all very closed to 200 which could indicate that after ~200 request the server wants you to open a new HTTP2 connection. I have seen reports of a similar behaviour now a couple of times. Can you say to which HTTP endpoint you are sending request to? Maybe we can find something in their documentation.

jake-betplusev commented

Thank you for the quick reply!

The StreamIDs are all very closed to 200 which could indicate that after ~200 request the server wants you to open a new HTTP2 connection.

This sounds very likely. I'm sending requests to multiple endpoints to get sports data. A lot of those endpoints have links in responses that I then traverse.

Dumb question: are you saying my server wants to open a new HTTP2 connection or an external server that I'm requesting from would like to open a new HTTP2 connection?

I currently use the client on the Application for all requests. In either case, would it resolve the issue by initializing new clients more often to limit the calls per client?

Also, why would this cause the ScheduledJob to fail instead of just making the API requests fail?

dnadoba commented

Dumb question: are you saying my server wants to open a new HTTP2 connection or an external server that I'm requesting from would like to open a new HTTP2 connection?

I currently use the client on the Application for all requests. In either case, would it resolve the issue by initializing new clients more often to limit the calls per client?

I think the server you are sending the HTTP requests to (through the HTTPClient) wants that after 200 request the HTTP/2 connection is closed and a new HTTP/2 connection is created. The most likely reason for that is server side load balancing. There is currently no easy way I can think of other than counting the request you are doing per origin (scheme, host and port tuple [and other minor stuff like tis config]).

Also, why would this cause the ScheduledJob to fail instead of just making the API requests fail?

Not familiar with that part of Vapor, sorry. Probably best to ask this kind of question in the vapor related repository or swift forum section.

jake-betplusev commented

I think the server you are sending the HTTP requests to (through the HTTPClient) wants that after 200 request the HTTP/2 connection is closed and a new HTTP/2 connection is created.

Thanks @dnadoba ! Another dumb q incoming -- how do you close an HTTP/2 connection and open a new one?

dnadoba commented

The only way currently possible with public API is by shutting down the whole HTTPClient by calling shutdown() on it.

jake-betplusev commented

Thanks @dnadoba ! I'll close this thread.