aws-beam/aws-elixir

S3.get_object with SSE-C signature error

Closed this issue · 7 comments

ryanwinchester commented

When I try to get an object with AWS.S3.get_object/22 with an object encrypted with SSE-C, I get a mismatched signature error.

Code:

def get_object(client, bucket, filename, encryption_key, opts \\ []) do
  sse_customer_algorithm = "AES256"
  sse_customer_key = Base.encode64(encryption_key)
  sse_customer_key_md5 = Crypto.hash(encryption_key, :md5) |> Base.encode64()

  range =
    case opts[:range] do
      nil -> nil
      {from, to} -> "bytes=#{from}-#{to}"
    end

  AWS.S3.get_object(
    client,
    bucket,
    filename,
    _part_number = nil,
    _response_cache_control = nil,
    _response_content_disposition = nil,
    _response_content_encoding = nil,
    _response_content_language = nil,
    _response_content_type = nil,
    _response_expires = nil,
    _version_id = nil,
    _expected_bucket_owner = nil,
    _if_match = nil,
    _if_modified_since = nil,
    _if_none_match = nil,
    _if_unmodified_since = nil,
    range,
    _request_payer = nil,
    sse_customer_algorithm,
    sse_customer_key,
    sse_customer_key_md5,
    _options = []
  )
end

Response:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>***redacted***</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256
20210403T192431Z
20210403/ca-central-1/s3/aws4_request
***redacted***</StringToSign>
<SignatureProvided>***redacted***</SignatureProvided>
<StringToSignBytes>***redacted***</StringToSignBytes>
<CanonicalRequest>GET
/my-bucket/09b6f2f2-020f-40d6-b66f-32cdb5561b95.zip

content-type:text/xml
host:s3.ca-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210403T192431Z
x-amz-server-side-encryption-customer-algorithm:AES256
x-amz-server-side-encryption-customer-key:***redacted***
x-amz-server-side-encryption-customer-key-md5:***redacted***

content-type;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption-customer-algorithm;x-amz-server-side-encryption-customer-key;x-amz-server-side-encryption-customer-key-md5
e3b0c442***redacted***b855</CanonicalRequest>
<CanonicalRequestBytes>...(truncated)

I have verified that I am using the same key as the upload.

I'm stumped...

philss commented

Hi @ryanwinchester 👋

I was able to reproduce the problem. I am investigating if there is any bugs related to the way we sign the request.
This may not be related with SSE-C but with the headers in the request signature process.

But just to gather more information, I would like to ask you some questions:

  • how did you upload the file before? with aws-elixir?
  • how did you generate your key?
    • if it was with openssl, can you provide the algorithm you are using?
  • did you use another library to perform the same task?
ryanwinchester commented 
* how did you upload the file before? with aws-elixir?

I used a modified version of the sample code in the Phoenix Live View docs, for the direct-to-S3 upload from browser:
https://hexdocs.pm/phoenix_live_view/uploads-external.html#direct-to-s3

Oh, actually, there was a dependency-free sample code earlier on. Looks like they added "using ExAws.S3" to the docs now for the actual signing part.

I have a modified version of the original dependency-free example, where I mainly just add the SSE-C stuff. I'll look into switching it out to use AWS.S3, though. Now that I have the dependency, I might as well use it.

* how did you generate your key?

For both upload and download I used Plug.Crypto.KeyGenerator.generate/2 with a passwordstring and a salt from :crypto.strong_rand_bytes(16)

* did you use another library to perform the same task?

I've got a download proof-of-concept I wrote and used ExAws, but I didn't try it with SSE-C.
I can try it tomorrow.

ryanwinchester commented

I have a modified version of the original dependency-free example, where I mainly just add the SSE-C stuff. I'll look into switching it out to use AWS.S3, though. Now that I have the dependency, I might as well use it.

Actually, I don't see a documented way to create a presigned URL or the signatures for browser HTTP POST submissions.

All I can find is the undocumented AWS.Request.sign_v4 for generating headers with signature.

I'm talking about:

I can maybe use the undocumented Request.sign_v4 for the headers needed in browser POST upload fields, and I don't need the download URL since I can't use it with SSE-C anyway.

Here is the current code I use for browser HTTP POST fields

def sign_form_upload(filename, opts) do
  config = Enum.into(config(), %{})
  bucket = config.bucket
  acl = "private"
  key = filename
  max_file_size = Keyword.fetch!(opts, :max_file_size)
  content_type = Keyword.fetch!(opts, :content_type)
  expires_in = Keyword.fetch!(opts, :expires_in)
  encryption_key = Keyword.fetch!(opts, :encryption_key)

  # This is already Base64 encoded, so we need to deal with that.
  encryption_key_md5 =
    encryption_key
    |> Base.decode64!()
    |> Crypto.hash(:md5)
    |> Base.encode64()

  expires_at = expires_at_from_expires_in(expires_in)
  amz_datetime = amz_datetime(expires_at)
  credential = credential(config, expires_at)

  encoded_policy =
    Base.encode64("""
    {
      "expiration": "#{DateTime.to_iso8601(expires_at)}",
      "conditions": [
        {"bucket": "#{bucket}"},
        ["eq", "$key", "#{key}"],
        {"acl": "#{acl}"},
        ["eq", "$Content-Type", "#{content_type}"],
        ["content-length-range", 0, #{max_file_size}],
        {"x-amz-server-side-encryption-customer-algorithm": "AES256"},
        {"x-amz-server-side-encryption-customer-key": "#{encryption_key}"},
        {"x-amz-server-side-encryption-customer-key-MD5": "#{encryption_key_md5}"},
        {"x-amz-credential": "#{credential}"},
        {"x-amz-algorithm": "AWS4-HMAC-SHA256"},
        {"x-amz-date": "#{amz_datetime}"}
      ]
    }
    """)

  fields = %{
    "key" => key,
    "acl" => acl,
    "content-type" => content_type,
    "x-amz-server-side-encryption-customer-algorithm" => "AES256",
    "x-amz-server-side-encryption-customer-key" => encryption_key,
    "x-amz-server-side-encryption-customer-key-MD5" => encryption_key_md5,
    "x-amz-credential" => credential,
    "x-amz-algorithm" => "AWS4-HMAC-SHA256",
    "x-amz-date" => amz_datetime,
    "policy" => encoded_policy,
    "x-amz-signature" => signature(config, expires_at, encoded_policy)
  }

  url = "https://#{bucket}.s3.amazonaws.com"

  {:ok, url, fields}
end
philss commented

@ryanwinchester thank you for all the information! 💜

I'm gonna try to reproduce the problem with SSE-C using another lib (ExAWS or something in another language).

philss commented

@ryanwinchester I finally found the bug!! 😃
It is fixed on master. Can you give it a try?

PS: You may need to change the dependency name from mix.exs to aws_elixir when pointing to GitHub. I'm not sure if this is definitive.

ryanwinchester commented

@ryanwinchester I finally found the bug!! 😃

Wow, good catch on the ordering stuff. I probably would not have caught that in a year.

It is fixed on master. Can you give it a try?

I got a file to download! 🎉

It won't unzip, but probably an unrelated issue.

PS: You may need to change the dependency name from mix.exs to aws_elixir when pointing to GitHub. I'm not sure if this is definitive.

I did need to.

Thanks so much!

  • 🎉1
  • ❤️1
philss commented

@ryanwinchester FYI: I had to revert the application name back to aws, so you need to update back in your mix.exs 😬

  • 😄1
  • 👀1