http4s/http4s-jetty

Jetty client sometimes uses chunked transfer encoding for GET requests

Opened this issue · 4 comments

bastewart commented

When sending a GET request (no body) using the Jetty client sometimes it sends multiple network-level requests and uses Transfer-Encoding: chunked. From local testing manually setting content-length to 0 means this issue does not occur.

I've tried to produce a test with TestContext from cats-effect but trying to use the Jetty client leads to a deadlock (I can create it and tear it down with no issues).

If I had to guess I would say this is due to a race between the Jetty request being sent and the content for that request being populated, the lines are here:

https://github.com/http4s/http4s/blob/30798ab62e4d4739249692e7e80a8831654aa28a/jetty-client/src/main/scala/org/http4s/client/jetty/JettyClient.scala#L28

If the request sends (jReq.send(rl)) before the delayed content has written (dcp.write(req), dcp is a StreamRequestContentProvider) the request will be chunked.

Possibly just inverting these lines might mean the race is less likely? Unless are that way around for a reason (send before write).

http4s version: 0.21.0-M5. Sorry, only just realised this isn't the latest but I think there have been no changes between versions.

travisbrown commented

I was looking at this and noticed a few things:

  • Moving the dcp.write action before the jReq.send seems to consistently give you the Content-Length and no Transfer-Encoding when there's no body.
  • Sleeping for a second after the jReq.send consistently gives you the Transfer-Encoding instead of Content-Length.
  • If you explicitly set the content length everything's fine either way.

For context we ran into this issue when talking to an akka-http server that gives an annoying "Sending an 2xx early" warning for any GET request with Transfer-Encoding: chunked.

bastewart commented

We have seen some errors in production with the 0 content-lenth "fix" that imply this is actually not a fix. An internal http4s "something terrible has happened" http4s error and a Jetty "incorrect content-length" error. In both cases the requests being sent a very simple GET requests.

These are very infrequent in percentage terms, but they do occur.

ERROR o.h.c.j.StreamRequestContentProvider - Unable to write to Jetty sinkjava.lang.Exception: something terrible has happened
	at org.http4s.client.jetty.StreamRequestContentProvider.$anonfun$pipe$3(StreamRequestContentProvider.scala:31)

ERROR o.h.client.jetty.ResponseListener - Failed requestorg.eclipse.jetty.http.BadMessageException: 500: Incorrect Content-Length 0!=46
	at org.eclipse.jetty.http.HttpGenerator.generateHeaders(HttpGenerator.java:632)
	at org.eclipse.jetty.http.HttpGenerator.generateRequest(HttpGenerator.java:236)
	at org.eclipse.jetty.client.http.HttpSenderOverHTTP$HeadersCallback.process(HttpSenderOverHTTP.java:214)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
	at org.eclipse.jetty.client.http.HttpSenderOverHTTP.sendHeaders(HttpSenderOverHTTP.java:62)
	at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:212)
	at org.eclipse.jetty.client.http.HttpChannelOverHTTP.send(HttpChannelOverHTTP.java:85)
	at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:128)
	at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:201)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP$Delegate.send(HttpConnectionOverHTTP.java:242)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.send(HttpConnectionOverHTTP.java:121)
	at org.eclipse.jetty.client.http.HttpDestinationOverHTTP.send(HttpDestinationOverHTTP.java:38)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:362)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:320)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:310)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:285)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
	at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:570)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:754)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:746)
	at org.http4s.client.jetty.JettyClient$.$anonfun$allocate$8(JettyClient.scala:29)
hamnis commented

do you use request bodies on those GET requests?

travisbrown commented

@hamnis No, it's just a simple .fetchAs[Json](GET(query)).