AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error
gabrielmancini opened this issue · 17 comments
Hello Folk's
Nice lib! :-)
i had an error when try the s3 partial_upload part
the code:
# ...
{:ok, %{
"InitiateMultipartUploadResult" => %{
"UploadId" => key
}}, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})
Stream.concat([head], rest)
|> Stream.chunk_every(chunk_size)
|> Enum.map(fn chunk ->
chunk_s =
chunk
|> Enum.join("\r\n")
size = :erlang.byte_size(chunk_s)
IO.inspect( size / @megabyte)
AWS.S3.upload_part(client, bucket, key, %{"Body" => chunk_s, "ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64()})
|> IO.inspect()
end)
AWS.S3.complete_multipart_upload(client, bucket, key, %{})error
{:error,
{:unexpected_response,
%{
body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>EDCQXBK7XAH5FRKH</RequestId><HostId>bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo=</HostId></Error>",
headers: [
{"x-amz-request-id", "EDCQXBK7XAH5FRKH"},
{"x-amz-id-2",
"bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo="},
{"allow", "HEAD, DELETE, GET, PUT"},
{"content-type", "application/xml"},
{"transfer-encoding", "chunked"},
{"date", "Mon, 26 Jul 2021 21:52:17 GMT"},
{"server", "AmazonS3"}
],
status_code: 405
}}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>EDCQXBK7XAH5FRKH</RequestId><HostId>bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo=</HostId></Error>", headers: [{"x-amz-request-id", "EDCQXBK7XAH5FRKH"}, {"x-amz-id-2", "bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 26 Jul 2021 21:52:17 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
(elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
(elixir 1.12.0) lib/stream.ex:649: Stream.run/1
some gotchas here?
Hi @gabrielmancini 👋
Just to confirm: is the error occurring inside the stream?
Also, have you tried to use the PartNumber parameter for the upload_part/4 call?
Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten. Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.
hello @philss thanks for the reply ;-)
i change a little bit the code but i had the same issue, pls take a look.
the code:
{:ok,
%{
"InitiateMultipartUploadResult" => %{
"UploadId" => key
}
}, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})
# the magic "must" happend! "bias detected"
Stream.concat([head], rest)
|> Stream.chunk_every(chunk_size)
|> Stream.with_index(1)
|> Enum.map(fn {chunk, i} ->
chunk_s =
chunk
|> Enum.join("\r\n")
size = :erlang.byte_size(chunk_s)
IO.inspect({i, size / @megabyte})
AWS.S3.upload_part(client, bucket, path, %{
"Body" => chunk_s,
"ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64(),
"PartNumber" => i,
"UploadId" => key
})
# every post returns 200 ok
|> IO.inspect()
end)
# https://github.com/aws-beam/aws-elixir/issues/84
AWS.S3.complete_multipart_upload(client, bucket, path, %{})
|> IO.inspect()the error:
{:error,
{:unexpected_response,
%{
body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>",
headers: [
{"x-amz-request-id", "RBWVRAPGGPPF746S"},
{"x-amz-id-2",
"z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="},
{"allow", "HEAD, DELETE, GET, PUT"},
{"content-type", "application/xml"},
{"transfer-encoding", "chunked"},
{"date", "Mon, 02 Aug 2021 15:48:06 GMT"},
{"server", "AmazonS3"}
],
status_code: 405
}}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>", headers: [{"x-amz-request-id", "RBWVRAPGGPPF746S"}, {"x-amz-id-2", "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 02 Aug 2021 15:48:06 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
(elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
(elixir 1.12.0) lib/stream.ex:649: Stream.run/1all the parts appers to work fine, all of then returns status code 200, like this output sample:
{:ok, nil,
%{
body: "",
headers: [
{"x-amz-id-2",
"6LK+sRvE0+hmCpMXepbMJIwN6SLhDntv50CKUNcj/fWqGvvBz38oD2kUhIYvgqs/6ZEw6DKEDNg="},
{"x-amz-request-id", "RBWYDXCATZ5XVHKC"},
{"date", "Mon, 02 Aug 2021 15:48:08 GMT"},
{"etag", "\"94efc8234db38870f667962c3425f5a6\""},
{"server", "AmazonS3"},
{"content-length", "0"}
],
status_code: 200
}}Hi there @gabrielmancini and @philss!
I'm currently facing the same issue. One thing that is missing from above is a proper input payload.
I am building it like this
parts =
Enum.map(uploads, fn {{:ok, etag}, index} ->
%{"ETag" => etag, "PartNumber" => index}
end)
input = %{"CompleteMultipartUpload" => parts, "uploadId" => upload_id}
AWS.S3.complete_multipart_upload(client, bucket, key, input)The error is the same though
"%{body: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>C31JVFSFYW0HXKPK</RequestId><HostId>n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=</HostId></Error>\", headers: [{\"x-amz-request-id\", \"C31JVFSFYW0HXKPK\"}, {\"x-amz-id-2\", \"n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=\"}, {\"Allow\", \"HEAD, DELETE, GET, PUT\"}, {\"Content-Type\", \"application/xml\"}, {\"Transfer-Encoding\", \"chunked\"}, {\"Date\", \"Thu, 24 Mar 2022 13:36:15 GMT\"}, {\"Server\", \"AmazonS3\"}, {\"Connection\", \"close\"}], status_code: 405}"
It looks like this endpoint does not allow POST requests (see the Allow response header) 🤔
This is indeed the culprit! Changing
aws-elixir/lib/aws/generated/s3.ex
Line 212 in a80272a
It is indeed specified as a POST in the spec document https://github.com/aws/aws-sdk-go/blob/a893727e4b5ed70693aa65bd1b081d4cdcd18a4b/models/apis/s3/2006-03-01/api-2.json#L33
👋 Not super familiar with the Elixir code but in aws-erlang this is a post and I know for a fact that this seems to work just fine judging by our S3 bucket containing complete gigantic files 🤔
So before going ahead and changing it, I'd like you (or someone else) to really scuba dive and ensure the bug isn't hiding elsewhere.
Hi @onno-vos-dev, thanks for joining in. I do think that this stinks.
@dmorn Oh I'm not arguing with you 😄 For inspiration, this is how our Input looks which judging by yours appears to be slightly different 🤔 Unless I'm more rusty in my Elixir than I think I am 🤔
Input =
#{<<"UploadId">> => UploadId,
<<"CompleteMultipartUpload">> =>
#{<<"Part">> =>
[#{<<"ETag">> => Etag, <<"PartNumber">> => PartNr} || {PartNr, Etag} <- PartEtags]}},Uh uh this is different indeed 😜 My erlang may be rusty now, it this what you're producing here?
%{
"CompleteMultipartUpload" => [
%{
"Part" => %{
"ETag" => "5592cc66124693a066c16198bc40e330",
"PartNumber" => 1
}
},
%{
"Part" => %{
"ETag" => "75afa18db5dfe9f747e39f03c2152c80",
"PartNumber" => 2
}
},
%{"Part" => %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}},
%{"Part" => %{...}},
%{...},
...
],
"uploadId" => "elided"
}The uploadId is lowercased here cause it is turned into a query parameter and should not be part of the payload (I double checked both the code and tried it with the uppercased version, I get a malformed input error from AWS)
Anyway it does not matter if I build it like above or like
%{
"CompleteMultipartUpload" => %{
"Part" => [
%{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
%{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
%{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
%{...},
...
]
},
"uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}The error persists 😅
I think your code produces the second payload I posted @onno-vos-dev right?
Anyway it does not matter if I build it like above or like
%{ "CompleteMultipartUpload" => %{ "Part" => [ %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1}, %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2}, %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}, %{...}, ... ] }, "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--" }The error persists sweat_smile
That looks similar to what I generate:
#{<<"CompleteMultipartUpload">> =>
#{<<"Part">> =>
[#{<<"ETag">> => <<"etag1">>,<<"PartNumber">> => 1},
#{<<"ETag">> => <<"etag2">>,<<"PartNumber">> => 2},
#{<<"ETag">> => <<"etag3">>,<<"PartNumber">> => 3}]},
<<"UploadId">> => <<"my-elixir-is-terrible">>}So the plot thickens 🤔 I need to do some thinking and digging.
I just double checked how ExAws works as we do multi-part uploads with that lib in another project. The input payload they produce resembles the first one of the last above and they do indeed POST.
OOOOk fot it @onno-vos-dev. This is all about the input payload as expected. After inspecting the encoded request body a found that a proper elixir input is built like
parts =
Enum.map(uploads, fn {{:ok, etag}, index} ->
%{"ETag" => etag, "PartNumber" => index}
end)
input = %{"CompleteMultipartUpload" => %{"Part" => parts}, "UploadId" => upload_id}
And should produce a payload that looks like
<CompleteMultipartUpload>
<Part>
<ETag>a2b0962b15f6d5716b3e9df437a8133b</ETag>
<PartNumber>1</PartNumber>
</Part>
<Part>
...
</Part>
</CompleteMultipartUpload>So that was it, bad input @gabrielmancini ! I believe this issue can be closed @philss, thank you again @onno-vos-dev! 🙌
Now I can put Elixir on my CV 😅 Glad to see this is resolved!
Ok if I close this issue?
@onno-vos-dev 😂 Yes I'm OK with it!