Unable to send request with body larger than 64Kb

[GRoguelon]

Hi,

I have faced an issue with the Req library when sending a request with a body greather than 65_535 bytes. I faced the issue passing the payload to AWS Lambda and I was able to reproduce it with a Mock service on Internet. Also it seems the issue only concerns HTTP/2 adapter.

The issue is related to Mint HTTP/2 option: INITIAL_WINDOW_SIZE where the default limit is: 65,535. It seems the only options is to pass a body as stream or iodata but the use of ExAws.Lambda is limiting my options, I've also tried to change the window size settings but without success.

FYI: I use Req as ExAws HTTP client adapter.

Steps to reproduce

It works:

iex(1)> Req.request(method: :post, url: "https://httpbin.org/post", body: :crypto.strong_rand_bytes(65536))
{:ok,
 %Req.Response{
   status: 200,
   headers: %{
     "access-control-allow-credentials" => ["true"],
     "access-control-allow-origin" => ["*"],
     "content-type" => ["application/json"],
     "date" => ["Tue, 20 Feb 2024 17:05:20 GMT"],
     "server" => ["gunicorn/19.9.0"]
   },
   body: %{
     "args" => %{},
     "data" => "data:application/octet-stream;base64,TRUNCATED",
     "files" => %{},
     "form" => %{},
     "headers" => %{
       "Accept-Encoding" => "gzip",
       "Content-Length" => "65536",
       "Host" => "httpbin.org",
       "User-Agent" => "req/0.4.11",
       "X-Amzn-Trace-Id" => "Root=1-65d4dbd0-6517ac7748ae3940701f65d4"
     },
     "json" => nil,
     "origin" => "201.103.95.210",
     "url" => "https://httpbin.org/post"
   },
   trailers: %{},
   private: %{}
 }}

It doens't work:

iex(2)> Req.request(method: :post, url: "https://httpbin.org/post", body: :crypto.strong_rand_bytes(65537))
{:error,
 %Mint.HTTPError{
   reason: {:exceeds_window_size, :request, 65535},
   module: Mint.HTTP2
 }}

Trying to change the settings and this (eventhough I feel I'm using it wrong):

iex(3)> Req.request(method: :post, url: "https://httpbin.org/post", body: :crypto.strong_rand_bytes(65537), connect_options: [client_settings: [initial_window_size: 68_000]])
{:error,
 %Mint.HTTPError{
   reason: {:exceeds_window_size, :request, 65536},
   module: Mint.HTTP2
 }}

Please let me know if you need additional information.

Regards,

[wojtekmach]

Thank you for the report, I’ll look into it soon. If you can drop down to using Finch directly and see the problem there too it’s most likely nothing I can do in Req.

[wojtekmach]

As a workaround you can also try forcing http/1: connect_options: [protocols: [:http1]]

[GRoguelon]

Thank you, I will try this. As it doesn't seem to be related to Req, should we close this issue?

[GRoguelon]

I confirm that switching to HTTP/1 is working:

iex(1)> Req.request(method: :post, url: "https://httpbin.org/post", body: :crypto.strong_rand_bytes(65537), connect_options: [protocols: [:http1]])
{:ok,
 %Req.Response{
   status: 200,
   headers: %{
     "access-control-allow-credentials" => ["true"],
     "access-control-allow-origin" => ["*"],
     "connection" => ["keep-alive"],
     "content-type" => ["application/json"],
     "date" => ["Tue, 20 Feb 2024 17:18:22 GMT"],
     "server" => ["gunicorn/19.9.0"]
   },
   body: %{
     "args" => %{},
     "data" => "data:application/octet-stream;base64,TRUNCATE",
     "files" => %{},
     "form" => %{},
     "headers" => %{
       "Accept-Encoding" => "gzip",
       "Content-Length" => "65537",
       "Host" => "httpbin.org",
       "User-Agent" => "req/0.4.11",
       "X-Amzn-Trace-Id" => "Root=1-65d4dede-6682bc935550847d57d9f18a"
     },
     "json" => nil,
     "origin" => "201.103.95.210",
     "url" => "https://httpbin.org/post"
   },
   trailers: %{},
   private: %{}
 }}

[wojtekmach]

I don’t know enough about HTTP/2 but if you can reproduce it in Finch please open up the issue there, it feels like something that should work out of the box there.

[GRoguelon]

@wojtekmach Hey sorry to reopen the issue, I was doing some testing with Finch in order to open the isuee there but something is wrong with Req.

If I'm doing this with the Finch instance created by Req, I have the error:

iex(1)> Finch.build(:post, "https://nghttp2.org/httpbin/post", [], :crypto.strong_rand_bytes(65538)) |> Finch.request(Req.Finch)
{:error,
 %Mint.HTTPError{
   reason: {:exceeds_window_size, :request, 65535},
   module: Mint.HTTP2
 }}

But if I create a Finch instance with the default options provided by Finch, it works:

iex(1)> {:ok, pid} = Finch.start_link(name: MyFinch, pools: %{ default: [ count: 1, size: 50 ] })
{:ok, #PID<0.635.0>}
iex(2)> Finch.build(:post, "https://nghttp2.org/httpbin/post", [], :crypto.strong_rand_bytes(65538)) |> Finch.request(MyFinch)
{:ok,
 %Finch.Response{
   status: 200,
   body: "{\n  \"args\": {}, \n  \"data\": \"data:application/octet-stream;base64,TRUNCATED",
   headers: [
     {"date", "Tue, 20 Feb 2024 18:01:14 GMT"},
     {"content-type", "application/json"},
     {"content-length", "87690"},
     {"access-control-allow-origin", "*"},
     {"access-control-allow-credentials", "true"},
     {"x-backend-header-rtt", "0.159641"},
     {"strict-transport-security", "max-age=31536000"},
     {"connection", "close"},
     {"alt-svc", "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600"},
     {"server", "nghttpx"},
     {"via", "1.1 nghttpx"},
     {"x-frame-options", "SAMEORIGIN"},
     {"x-xss-protection", "1; mode=block"},
     {"x-content-type-options", "nosniff"}
   ],
   trailers: []
 }}

With HTTP/2 protocol only, it works:

iex(1)> {:ok, pid} = Finch.start_link(name: MyFinch, pools: %{ default: [ protocols: [:http2] ] })
{:ok, #PID<0.635.0>}
iex(2)> Finch.build(:post, "https://nghttp2.org/httpbin/post", [], :crypto.strong_rand_bytes(65538)) |> Finch.request(MyFinch)
{:ok,
 %Finch.Response{
   status: 200,
   body: "{\n  \"args\": {}, \n  \"data\": \"data:application/octet-stream;base64,TRUNCATED" <> ...,
   headers: [
     {"date", "Tue, 20 Feb 2024 18:03:36 GMT"},
     {"content-type", "application/json"},
     {"content-length", "87690"},
     {"access-control-allow-origin", "*"},
     {"access-control-allow-credentials", "true"},
     {"x-backend-header-rtt", "0.170131"},
     {"strict-transport-security", "max-age=31536000"},
     {"connection", "close"},
     {"alt-svc", "h3=\":443\"; ma=3600, h3-29=\":443\"; ma=3600"},
     {"server", "nghttpx"},
     {"via", "1.1 nghttpx"},
     {"x-frame-options", "SAMEORIGIN"},
     {"x-xss-protection", "1; mode=block"},
     {"x-content-type-options", "nosniff"}
   ],
   trailers: []
 }}

It seems the default option protocols: [:http1, http2] is causing the issue, based on the documentation, I'm wondering (I'm not an expert) if it isn't related to the Multiplexing which is disabled when HTTP/1 is provided as supported protocol.

Quote from Finch.start_link/1:

If using :http1 only, an HTTP1 pool without multiplexing is used. If using :http2 only, an HTTP2 pool with multiplexing is used. If both are listed, then both HTTP1/HTTP2 connections are supported (via ALPN), but there is no multiplexing.

Would be an option to have an Elixir configuration to override the option passed to the start of Req.Finch?

Thank you

[wojtekmach]

Oh so if you pass connect_options: [protocols: :http2]] in Req it works?

[GRoguelon]

Actually, yes:

Req.post(url: "https://nghttp2.org/httpbin/post", body: :crypto.strong_rand_bytes(65538), connect_options: [protocols: [:http2]])

[GRoguelon]

@wojtekmach Do you think it should be documented somewhere?

[wojtekmach]

Ok, great find. Could you open an issue on Finch after all? If the http 1/2 negotiation there is broken we should address it there.

[GRoguelon]

I created the issue there. Thank you for your assistance.

[kzemek]

@wojtekmach should the default connect_options for finch be reverted to pre-4.10 state until it's fixed? As it stands, req is by default all but broken for large requests to H2 hosts.

[wojtekmach]

@kzemek yup, I released v0.4.13 which uses the old default: protocols: [:http1] for now. Thanks!