SignatureDoesNotMatch for S3 key with space

[webengineer]

Environment

• Elixir & Erlang versions (elixir --version):

Erlang/OTP 23 [erts-11.0.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.10.4 (compiled with Erlang/OTP 23)

• ExAws version mix deps |grep ex_aws

* ex_aws 2.1.4 (Hex package) (mix)
  locked at 2.1.4 (ex_aws) 18eae006
* ex_aws_s3 2.0.2 (Hex package) (mix)
  locked at 2.0.2 (ex_aws_s3) 0569f5b2

• HTTP client version. IE for hackney do mix deps | grep hackney

* hackney 1.16.0 (Hex package) (rebar3)
  locked at 1.16.0 (hackney) 3bf0bebb

Current behavior

ExAws.S3.list_objects("some-bucket", marker: "key_WITH SPACE") |> ExAws.request()
{:error,
 {:http_error, 403,
  %{
    body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

This may be fixed with

ExAws.S3.list_objects("some-bucket", marker: URI.encode("key_WITH SPACE")) |> ExAws.request()

BUT the issue happens also when during streaming (ExAws.stream!) the continuation marker has spaces. The marker is processed internally by the ExAws.S3 and can't be encoded.

Expected behavior

ExAws.S3.list_objects("some-bucket", marker: "key_WITH SPACE") |> ExAws.request()
{:ok, 

[cjbottaro]

I tracked this down to this snippet in ex_aws/lib/ex_aws/request.ex:

  def request_and_retry(method, url, service, config, headers, req_body, {:attempt, attempt}) do
    full_headers = ExAws.Auth.headers(method, url, service, config, headers, req_body)

    with {:ok, full_headers} <- full_headers do
      safe_url = ExAws.Request.Url.sanitize(url, service)

It's the ExAws.Request.Url.sanitize(url, service) which is causing problems. If you don't call that, then everything works fine.

This may be fixed with

ExAws.S3.list_objects("some-bucket", marker: URI.encode("key_WITH SPACE")) |> ExAws.request()

That will make the request successful, but you might get incomplete results. That's how we originally tried working around the problem, but thankfully we tested against the Ruby aws-sdk-v3 gem... and the Ruby library returned 4 more results (out of ~22k) when scanning through object versions. It took hours of debugging to finally realize the cause of the discrepancy.

I think this is a pretty bad bug. It doesn't crash, but leads to incorrect results that people might not even notice.

[bernardd]

@cjbottaro Thanks heaps for the info. My concern is that presumably the sanitize call is there for a reason (albeit evidently one that doesn't apply in your case), and maybe there's just some tweak to be made inside it. Are you able to show the output of the debug line below that call (the one that starts "ExAws: Request URL: for the working and non-working cases? If we can pin down what needs to be tweaked in sanitize that would make me feel much better.