Support for S3 Compatible APIs
collegeimprovements opened this issue · 12 comments
We use Linode's Object Storage. And it's fully compatible with S3.
When we try to use it with aws-elixir
we get the following error.
iex(1)> client = AWS.Client.create("Access-Key", "Secret-Key", "us-east-1") |> Map.put(:endpoint, "us-east-1.linodeobjects.com")
%AWS.Client{
access_key_id: "access-key",
endpoint: "us-east-1.linodeobjects.com",
http_client: {AWS.HTTPClient, []},
json_module: {AWS.JSON, []},
port: 443,
proto: "https",
region: "us-east-1",
secret_access_key: "secret-key",
service: nil,
session_token: nil,
xml_module: {AWS.XML, []}
}
iex(2)> AWS.S3.list_buckets(client)
[info] TLS :client: In state :certify at ssl_handshake.erl:1901 generated CLIENT ALERT: Fatal - Handshake Failure
- {:bad_cert, :hostname_check_failed}
{:error,
{:tls_alert,
{:handshake_failure,
'TLS client: In state certify at ssl_handshake.erl:1901 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}'}}}
S3cmd
and other tools work fine with S3-related operations with Linode's storage: Docs.
Would this issue be in scope of this project ?
This isn't a problem with the aws elixir library per se, it's a problem with the TLS certificate validation step that happens before the library sends commands to the API endpoint. So what release of Elixir and Erlang are you using and how old your TLS certificates?
I'm on latest version of Elixir and Erlang.
Elixir: 1.12.2-otp-24
Erlang: 24.0.3
λ › elixir -v
Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]
Elixir 1.12.2 (compiled with Erlang/OTP 24)
OK, and whatever HTTP client you're using has an up-to-date certificate bundle? What about the remote endpoint?
IIRC, OTP 24 refuses to talk to older TLS implementations by default.
@collegeimprovements @mrallen1 I think we still don't support setting the endpoint directly in the client. This is similar to #85
The reason for the error may be related to the AWS.Resquest
overwriting the endpoint value.
We are facing the same issue. Is there any workaround available now?
@collegeimprovements @zakimedina can you check this PR if it fixes your issue? https://github.com/aws-beam/aws-elixir/pull/91/files#diff-5a31358e99eb904d8d9019d153ff5dec3f14890d446a070ec1010934c71fb910R125
@collegeimprovements thank you for checking that :)
And yeah, pre signed urls are possible with another package: https://hex.pm/packages/aws_signature
Please check: https://hexdocs.pm/aws_signature/aws_signature.html#sign_v4_query_params/7
Thanks a lot @philss. That works great.
Here is the function that we have for signed_url
.
It may save some time for someone looking into it.
def get_signed_url(full_object_path, opts \\ [ttl: 86_400]) do
access_key = System.get_env("S3_ACCESS_KEY", "default_access_key")
secret_key = System.get_env("S3_SECRET_KEY", "default_secret_key")
region = System.get_env("S3_REGION", "us-east-1")
datetime = :erlang.universaltime()
base_url =
System.get_env("S3_BASE_URL", "https://#{bucket_name}.us-east-1.linodeobjects.com")
url = "#{base_url}/#{full_object_path}"
signed_url =
:aws_signature.sign_v4_query_params(
access_key,
secret_key,
region,
"S3",
datetime,
url,
opts
)
signed_url
end
@collegeimprovements I think Jonatan's fix may solve the problem. Please take a look there.
@philss Yes, it indeed fixed the issue. Thanks a lot :)