[feature] Support compression with Basic HTTP Client
Opened this issue · 3 comments
Version and OS
v0.47.03 on linux/docker
Is your feature request related to a problem? Please describe.
I had to poll this URL recently: https://www.revolve.com/never-fully-dressed-atlantis-cardigan-in-blue/dp/NEVR-WK3/?size=M&code=NEVR-WK3
I observed that it only works with curl if I have a Chrome user agent and --compressed
Describe the solution you'd like
I don't think that I can enable compression with the basic HTTP agent, which is why my polling is failing, although I don't have conclusive evidence of this just that the curl works and the cd.io poll fails. I'd be very happy to get further debug if you tell me what you'd like to see.
Describe the use-case and give concrete real-world examples
Attach any HTML/JSON, give links to sites, screenshots etc, we are not mind readers
Additional context
Add any other context or screenshots about the feature request here.
hmmm requests supports deflate so im wondering which exact compression algorithm it needs?
$ curl "https://www.revolve.com/never-fully-dressed-atlantis-cardigan-in-blue/dp/NEVR-WK3/?size=M&code=NEVR-WK3"
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
dgtlmoon@dgtlmoon-HP-ProBook-440-G7:~$
actually this is related to #2763 http-2 support
< content-encoding: gzip
It'll work if you curl it like this:
curl --compressed -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" "https://www.revolve.com/never-fully-dressed-atlantis-cardigan-in-blue/dp/NEVR-WK3/?size=M&code=NEVR-WK3"