failed to proxy a chunked response from upstream
Closed this issue · 4 comments
If upstream responded a chunked response (i.e. transfer-encoding: chunked in response header but without content-length header), the proxy would remove the header (as it's a hop-by-hop header, see #6), then the response that was sent to the recipient did not include a content-length header, which resulted to an error when processing the response in the recipient.
Do you think this is an issue and any ideas on how this should be handled?
Meanwhile, I am trying to understand how this is handled in other proxies.
very interesting read about what a proxy should do for chunked request/response: https://www.mnot.net/blog/2011/07/11/what_proxies_must_do
in jetty-proxy, I can see the transfer-encoding header is also stripped, and the response is likely sent to recipient still in chunks? see https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/jetty-proxy/src/main/java/org/eclipse/jetty/proxy/ProxyServlet.java#L220, don't fully get it though 🤔
Yes agree with the blog entry and the intent is that each hop is its own thing re. whether to chunk the content.
So, the server that the proxy is calling chooses to send the transfer-encoding: chunked header and not set a content-length. Your proxy might, say, need to see the whole message in order to perform some function before returning; as such, you may have read the entire message into memory (or quit as you approach a configured maximum response size). At that point, you want to send that response back to your client... now you know the content length so it's rather up to you as to whether you don't bother with the chunked transfer and set a content length or to reproduce what the server did. So long as each hop is consistent, the client shouldn't get confused.
I have also seen certain scenarios where a proxy is protecting a back-end service and you want to reject chunked transfers as they are a known vector for denial of service attacks (e.g. open a stream and keep writing to see if you can swamp the other end. That's another story though.
@EdgewareRoad Thanks for the reply, it makes sense. now it looks like I need to override copyResponseEntity() method to handle it for my case.
Closing the issue for now.