AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error
gabrielmancini opened this issue ยท 17 comments
Hello Folk's
Nice lib! :-)
i had an error when try the s3 partial_upload part
the code:
# ...
{:ok, %{
"InitiateMultipartUploadResult" => %{
"UploadId" => key
}}, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})
Stream.concat([head], rest)
|> Stream.chunk_every(chunk_size)
|> Enum.map(fn chunk ->
chunk_s =
chunk
|> Enum.join("\r\n")
size = :erlang.byte_size(chunk_s)
IO.inspect( size / @megabyte)
AWS.S3.upload_part(client, bucket, key, %{"Body" => chunk_s, "ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64()})
|> IO.inspect()
end)
AWS.S3.complete_multipart_upload(client, bucket, key, %{})
error
{:error,
{:unexpected_response,
%{
body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>EDCQXBK7XAH5FRKH</RequestId><HostId>bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo=</HostId></Error>",
headers: [
{"x-amz-request-id", "EDCQXBK7XAH5FRKH"},
{"x-amz-id-2",
"bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo="},
{"allow", "HEAD, DELETE, GET, PUT"},
{"content-type", "application/xml"},
{"transfer-encoding", "chunked"},
{"date", "Mon, 26 Jul 2021 21:52:17 GMT"},
{"server", "AmazonS3"}
],
status_code: 405
}}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>EDCQXBK7XAH5FRKH</RequestId><HostId>bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo=</HostId></Error>", headers: [{"x-amz-request-id", "EDCQXBK7XAH5FRKH"}, {"x-amz-id-2", "bD0jm08lboSt4yjEOsNxXTrxGb/i3rBdNATN+uD+mgTZ+F0w64aFddXj6Vthcrc4ClD8DBX0buo="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 26 Jul 2021 21:52:17 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
(elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
(elixir 1.12.0) lib/stream.ex:649: Stream.run/1
some gotchas here?
Hi @gabrielmancini ๐
Just to confirm: is the error occurring inside the stream?
Also, have you tried to use the PartNumber
parameter for the upload_part/4
call?
Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten. Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.
hello @philss thanks for the reply ;-)
i change a little bit the code but i had the same issue, pls take a look.
the code:
{:ok,
%{
"InitiateMultipartUploadResult" => %{
"UploadId" => key
}
}, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})
# the magic "must" happend! "bias detected"
Stream.concat([head], rest)
|> Stream.chunk_every(chunk_size)
|> Stream.with_index(1)
|> Enum.map(fn {chunk, i} ->
chunk_s =
chunk
|> Enum.join("\r\n")
size = :erlang.byte_size(chunk_s)
IO.inspect({i, size / @megabyte})
AWS.S3.upload_part(client, bucket, path, %{
"Body" => chunk_s,
"ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64(),
"PartNumber" => i,
"UploadId" => key
})
# every post returns 200 ok
|> IO.inspect()
end)
# https://github.com/aws-beam/aws-elixir/issues/84
AWS.S3.complete_multipart_upload(client, bucket, path, %{})
|> IO.inspect()
the error:
{:error,
{:unexpected_response,
%{
body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>",
headers: [
{"x-amz-request-id", "RBWVRAPGGPPF746S"},
{"x-amz-id-2",
"z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="},
{"allow", "HEAD, DELETE, GET, PUT"},
{"content-type", "application/xml"},
{"transfer-encoding", "chunked"},
{"date", "Mon, 02 Aug 2021 15:48:06 GMT"},
{"server", "AmazonS3"}
],
status_code: 405
}}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>", headers: [{"x-amz-request-id", "RBWVRAPGGPPF746S"}, {"x-amz-id-2", "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 02 Aug 2021 15:48:06 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
(elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
(elixir 1.12.0) lib/stream.ex:649: Stream.run/1
all the parts appers to work fine, all of then returns status code 200, like this output sample:
{:ok, nil,
%{
body: "",
headers: [
{"x-amz-id-2",
"6LK+sRvE0+hmCpMXepbMJIwN6SLhDntv50CKUNcj/fWqGvvBz38oD2kUhIYvgqs/6ZEw6DKEDNg="},
{"x-amz-request-id", "RBWYDXCATZ5XVHKC"},
{"date", "Mon, 02 Aug 2021 15:48:08 GMT"},
{"etag", "\"94efc8234db38870f667962c3425f5a6\""},
{"server", "AmazonS3"},
{"content-length", "0"}
],
status_code: 200
}}
Hi there @gabrielmancini and @philss!
I'm currently facing the same issue. One thing that is missing from above is a proper input payload.
I am building it like this
parts =
Enum.map(uploads, fn {{:ok, etag}, index} ->
%{"ETag" => etag, "PartNumber" => index}
end)
input = %{"CompleteMultipartUpload" => parts, "uploadId" => upload_id}
AWS.S3.complete_multipart_upload(client, bucket, key, input)
The error is the same though
"%{body: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>C31JVFSFYW0HXKPK</RequestId><HostId>n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=</HostId></Error>\", headers: [{\"x-amz-request-id\", \"C31JVFSFYW0HXKPK\"}, {\"x-amz-id-2\", \"n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=\"}, {\"Allow\", \"HEAD, DELETE, GET, PUT\"}, {\"Content-Type\", \"application/xml\"}, {\"Transfer-Encoding\", \"chunked\"}, {\"Date\", \"Thu, 24 Mar 2022 13:36:15 GMT\"}, {\"Server\", \"AmazonS3\"}, {\"Connection\", \"close\"}], status_code: 405}"
It looks like this endpoint does not allow POST requests (see the Allow response header) ๐ค
This is indeed the culprit! Changing
aws-elixir/lib/aws/generated/s3.ex
Line 212 in a80272a
It is indeed specified as a POST in the spec document https://github.com/aws/aws-sdk-go/blob/a893727e4b5ed70693aa65bd1b081d4cdcd18a4b/models/apis/s3/2006-03-01/api-2.json#L33
๐ Not super familiar with the Elixir code but in aws-erlang this is a post
and I know for a fact that this seems to work just fine judging by our S3 bucket containing complete gigantic files ๐ค
So before going ahead and changing it, I'd like you (or someone else) to really scuba dive and ensure the bug isn't hiding elsewhere.
Hi @onno-vos-dev, thanks for joining in. I do think that this stinks.
@dmorn Oh I'm not arguing with you ๐ For inspiration, this is how our Input
looks which judging by yours appears to be slightly different ๐ค Unless I'm more rusty in my Elixir than I think I am ๐ค
Input =
#{<<"UploadId">> => UploadId,
<<"CompleteMultipartUpload">> =>
#{<<"Part">> =>
[#{<<"ETag">> => Etag, <<"PartNumber">> => PartNr} || {PartNr, Etag} <- PartEtags]}},
Uh uh this is different indeed ๐ My erlang may be rusty now, it this what you're producing here?
%{
"CompleteMultipartUpload" => [
%{
"Part" => %{
"ETag" => "5592cc66124693a066c16198bc40e330",
"PartNumber" => 1
}
},
%{
"Part" => %{
"ETag" => "75afa18db5dfe9f747e39f03c2152c80",
"PartNumber" => 2
}
},
%{"Part" => %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}},
%{"Part" => %{...}},
%{...},
...
],
"uploadId" => "elided"
}
The uploadId is lowercased here cause it is turned into a query parameter and should not be part of the payload (I double checked both the code and tried it with the uppercased version, I get a malformed input error from AWS)
Anyway it does not matter if I build it like above or like
%{
"CompleteMultipartUpload" => %{
"Part" => [
%{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
%{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
%{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
%{...},
...
]
},
"uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}
The error persists ๐
I think your code produces the second payload I posted @onno-vos-dev right?
Anyway it does not matter if I build it like above or like
%{ "CompleteMultipartUpload" => %{ "Part" => [ %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1}, %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2}, %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}, %{...}, ... ] }, "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--" }The error persists sweat_smile
That looks similar to what I generate:
#{<<"CompleteMultipartUpload">> =>
#{<<"Part">> =>
[#{<<"ETag">> => <<"etag1">>,<<"PartNumber">> => 1},
#{<<"ETag">> => <<"etag2">>,<<"PartNumber">> => 2},
#{<<"ETag">> => <<"etag3">>,<<"PartNumber">> => 3}]},
<<"UploadId">> => <<"my-elixir-is-terrible">>}
So the plot thickens ๐ค I need to do some thinking and digging.
I just double checked how ExAws works as we do multi-part uploads with that lib in another project. The input payload they produce resembles the first one of the last above and they do indeed POST.
OOOOk fot it @onno-vos-dev. This is all about the input payload as expected. After inspecting the encoded request body a found that a proper elixir input is built like
parts =
Enum.map(uploads, fn {{:ok, etag}, index} ->
%{"ETag" => etag, "PartNumber" => index}
end)
input = %{"CompleteMultipartUpload" => %{"Part" => parts}, "UploadId" => upload_id}
And should produce a payload that looks like
<CompleteMultipartUpload>
<Part>
<ETag>a2b0962b15f6d5716b3e9df437a8133b</ETag>
<PartNumber>1</PartNumber>
</Part>
<Part>
...
</Part>
</CompleteMultipartUpload>
So that was it, bad input @gabrielmancini ! I believe this issue can be closed @philss, thank you again @onno-vos-dev! ๐
Now I can put Elixir on my CV ๐ Glad to see this is resolved!
Ok if I close this issue?
@onno-vos-dev ๐ Yes I'm OK with it!