aws-beam/aws-elixir

Support for S3 Compatible APIs

collegeimprovements opened this issue · 12 comments

collegeimprovements commented

We use Linode's Object Storage. And it's fully compatible with S3.
When we try to use it with aws-elixir we get the following error.

iex(1)> client = AWS.Client.create("Access-Key", "Secret-Key", "us-east-1") |> Map.put(:endpoint, "us-east-1.linodeobjects.com") 
%AWS.Client{
  access_key_id: "access-key",
  endpoint: "us-east-1.linodeobjects.com",
  http_client: {AWS.HTTPClient, []},
  json_module: {AWS.JSON, []},
  port: 443,
  proto: "https",
  region: "us-east-1",
  secret_access_key: "secret-key",
  service: nil,
  session_token: nil,
  xml_module: {AWS.XML, []}
}

iex(2)> AWS.S3.list_buckets(client)
[info] TLS :client: In state :certify at ssl_handshake.erl:1901 generated CLIENT ALERT: Fatal - Handshake Failure
 - {:bad_cert, :hostname_check_failed}
{:error,
 {:tls_alert,
  {:handshake_failure,
   'TLS client: In state certify at ssl_handshake.erl:1901 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}'}}}

S3cmd and other tools work fine with S3-related operations with Linode's storage: Docs.

Would this issue be in scope of this project ?

jadeallenx commented

This isn't a problem with the aws elixir library per se, it's a problem with the TLS certificate validation step that happens before the library sends commands to the API endpoint. So what release of Elixir and Erlang are you using and how old your TLS certificates?

collegeimprovements commented

I'm on latest version of Elixir and Erlang.

Elixir: 1.12.2-otp-24
Erlang: 24.0.3

λ › elixir -v
Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.12.2 (compiled with Erlang/OTP 24)
jadeallenx commented

OK, and whatever HTTP client you're using has an up-to-date certificate bundle? What about the remote endpoint?

IIRC, OTP 24 refuses to talk to older TLS implementations by default.

philss commented

@collegeimprovements @mrallen1 I think we still don't support setting the endpoint directly in the client. This is similar to #85
The reason for the error may be related to the AWS.Resquest overwriting the endpoint value.

zakimedina commented

We are facing the same issue. Is there any workaround available now?

collegeimprovements commented

Hi @philss, It works 😃 Thanks a lot ☺️

I checked, list_buckets, get_object and put_object. All of them work like a charm.

What is the recommended way to generate pre_signed_urls ?
#87

philss commented

@collegeimprovements thank you for checking that :)

And yeah, pre signed urls are possible with another package: https://hex.pm/packages/aws_signature
Please check: https://hexdocs.pm/aws_signature/aws_signature.html#sign_v4_query_params/7

  • ❤️1
collegeimprovements commented

Thanks a lot @philss. That works great.

Here is the function that we have for signed_url.
It may save some time for someone looking into it.

  def get_signed_url(full_object_path, opts \\ [ttl: 86_400]) do
    access_key = System.get_env("S3_ACCESS_KEY", "default_access_key")
    secret_key = System.get_env("S3_SECRET_KEY", "default_secret_key")
    region = System.get_env("S3_REGION", "us-east-1")
    datetime = :erlang.universaltime()

    base_url =
      System.get_env("S3_BASE_URL", "https://#{bucket_name}.us-east-1.linodeobjects.com")

    url = "#{base_url}/#{full_object_path}"

    signed_url =
      :aws_signature.sign_v4_query_params(
        access_key,
        secret_key,
        region,
        "S3",
        datetime,
        url,
        opts
      )

    signed_url
  end
  • ❤️2
collegeimprovements commented

Hi @philss,

Everything works well from aws-elixir point of view.
Though we are not able to upload a file from the signed-url.

I've created an issue on aws-signature repo.
But I'm not sure what's wrong as the generated url works to access/download files.

philss commented

@collegeimprovements I think Jonatan's fix may solve the problem. Please take a look there.

collegeimprovements commented

@philss Yes, it indeed fixed the issue. Thanks a lot :)

  • ❤️1