Streaming bodies sometimes never produce a response without Transfer-Encoding set

[rzeigler]

Using blaze server, when rendering a streaming body directly using withBodyStream if the Transfer-Encoding: chunked header is omitted no response is ever rendered to client in some cases. This looks like the result of the idle timeout being hit before the response body is finished.

For a reproduction see a repro built on the http4s.g8 template as of 10/7/2020.
Specifically this route happily streams small chunks to curl whereas this other route never sends any response to the client before timeout.

The server itself is built with idle timeout of 5 seconds.

To see the behavior from the perspective of curl, see which is also in the root of the repo.

edits: embed the asciinema directly

[vasilmkd]

I'm interested to investigate this one, but I have a question. Can this really be considered a bug or just a misconfiguration on the part of the user? What should the server do when the header is missing?

[vasilmkd]

I'm looking at the definition of withBodyStream itself and to me it seems like wrong headers might be an issue.

  /** Sets the entity body without affecting headers such as `Transfer-Encoding`
    * or `Content-Length`. Most use cases are better served by [[withEntity]],
    * which uses an [[EntityEncoder]] to maintain the headers.
    */
  def withBodyStream(body: EntityBody[F]): Self =
    change(body = body)

[rossabaker]

Per the HTTP spec, these messages have no body. Per http4s tradition, we try to render a body using either a Content-Length (if we know we've buffered the whole thing before first flush) or a chunked transfer encoding.

When http4s/http4s#4509 is done, we can stop guessing, and this gets cleaner. But I'd expect the backend to infer a header. CachingChunkWriter in blaze is one implementation.