Bug: SQS send_batch_message error encoding list of Entries

[homanchou]

I'm looking at the docs for https://hexdocs.pm/aws/AWS.SQS.html#send_message_batch/3

send_message_batch(client, input, options \\ [])

but it's not clear what the input parameter is. Looking at the aws sdk docs for javascript and golang, I'm inferring that input is something like this in elixir:

input = %{
          "Entries" => batch, #list,
          "QueueUrl" => "..." #string
        }

Where the Entries key should probably be something like a list of maps like this:

 batch = [%{
      "Id" => id,
      "MessageBody" => message_body,
      "MessageDeduplicationId" => group_id,
      "MessageGroupId" => group_id
  }]

The problem is, when I call:

AWS.SQS.send_message_batch(client, input)

I get the error:

** (ArgumentError) cannot convert the given list to a string.

To be converted to a string, a list must either be empty or only
contain the following elements:

  * strings
  * integers representing Unicode code points
  * a list containing one of these three elements

Is this a bug or am I doing something wrong?

It seems as though "Entries" can't be a list of maps, but I don't know what else to give it. I expect to be able to use a similar api as the javascript and golang usage.

[philss]

@homanchou would you mind to send the backtrace of the error?

For what I could understand, there is a problem in the "querystring encoder" that is not supporting complex values for the input.

but it's not clear what the input parameter is. Looking at the aws sdk docs for javascript and golang, I'm inferring that input is something like this in elixir:

You are right.

Perhaps what we need is to encode the map as JSON, or perhaps another format. Can you try to send the input like this?:

input = %{
          "Entries" => Jason.encode!(batch),
          "QueueUrl" => "..." #string
        }

[homanchou]

Here is more of the error:

[error] GenServer Thexr.EventWriter terminating
** (ArgumentError) cannot convert the given list to a string.

To be converted to a string, a list must either be empty or only
contain the following elements:

  * strings
  * integers representing Unicode code points
  * a list containing one of these three elements

Please check the given list or call inspect/1 to get the list representation, got:

[%{"Id" => "dc759dc9-8df4-40cc-8957-5b7fba160e29#2483", "MessageBody" => "{\"event_timestamp\":1653675595058,\"payload\":{\"member_id\":\"rCowHT\",\"msg\":\"no hands\"},\"sequence\":2483,\"space_id\":\"dc759dc9-8df4-40cc-8957-5b7fba160e29\",\"type\":\"message_broadcasted\"}", "MessageDeduplicationId" => "dc759dc9-8df4-40cc-8957-5b7fba160e29", "MessageGroupId" => "dc759dc9-8df4-40cc-8957-5b7fba160e29"}]

    (elixir 1.13.1) lib/list.ex:978: List.to_string/1
    (aws 0.11.0) lib/aws/util.ex:27: anonymous fn/1 in AWS.Util.encode_query/1
    (elixir 1.13.1) lib/enum.ex:1597: anonymous fn/3 in Enum.map/2
    (stdlib 3.17) maps.erl:410: :maps.fold_1/3
    (elixir 1.13.1) lib/enum.ex:2408: Enum.map/2
    (aws 0.11.0) lib/aws/util.ex:27: AWS.Util.encode_query/1
    (aws 0.11.0) lib/aws/request.ex:42: AWS.Request.request_post/5
    (thexr 0.0.1) lib/thexr/event_writer.ex:64: anonymous fn/2 in Thexr.EventWriter.handle_events/3
    (elixir 1.13.1) lib/enum.ex:937: Enum."-each/2-lists^foreach/1-0-"/2
    (thexr 0.0.1) lib/thexr/event_writer.ex:55: Thexr.EventWriter.handle_events/3
    (gen_stage 1.1.2) lib/gen_stage.ex:2471: GenStage.consumer_dispatch/6
    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4
    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6
    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_consumer", {#PID<0.667.0>, #Reference<0.3001471844.3051880450.127173>}, [%{event_timestamp: 1653675595058, payload: %{member_id: "rCowHT", msg: "no hands"}, sequence: 2483, space_id: "dc759dc9-8df4-40cc-8957-5b7fba160e29", type: "message_broadcasted"}]}
State: %{aws_client: #AWS.Client<endpoint: nil, http_client: {AWS.HTTPClient, []}, json_module: {AWS.JSON, []}, port: 443, proto: "https", region: "us-west-2", service: nil, xml_module: {AWS.XML, []}, ...>, env_name: :dev, sqs_url: "https://sqs.us-west-2.amazonaws.com/426932470747/ThexrEventQueue.fifo"}

Can you try to send the input like this?:

If I try to Jason.encode! the "Entries" as you recommended, then I get this response error from AWS:

{:error,
 {:unexpected_response,
  %{
    body: "<?xml version=\"1.0\"?><ErrorResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><Error><Type>Sender</Type><Code>MalformedInput</Code><Message>Unexpected list element termination</Message><Detail/></Error><RequestId>aff96c21-1888-5c00-9fd5-372a013fa48f</RequestId></ErrorResponse>",
    headers: [
      {"x-amzn-RequestId", "aff96c21-1888-5c00-9fd5-372a013fa48f"},
      {"Date", "Fri, 27 May 2022 18:24:55 GMT"},
      {"Content-Type", "text/xml"},
      {"Content-Length", "286"}
    ],
    status_code: 400
  }}}

[philss]

@homanchou thank you. I will take a look this week. It's a bug in the encoding of query strings.

[philss]

@homanchou I fixed this in #142

Also, I think your payload is wrong. According to the SendMessageBatch docs, you need to pass the entries with the "SendMessageBatchRequestEntry" key (not "Entries").

The input map will look like this:

input = %{
  "QueueUrl" => "[YOUR-QUEUE-URL]",
  "SendMessageBatchRequestEntry" => [
    %{"Id" => "082cf3ff", "MessageBody" => "this is a test"},
    %{"Id" => "30181acb", "MessageBody" => "this is another test"}
  ]
}

To send the message you will do:

AWS.SQS.send_message_batch(client, input)

Just another tip: I could only receive the messages when I included the "AttributeName" attribute:

AWS.SQS.receive_message(client, %{"QueueUrl" => queue, "AttributeName" => "All"})

[philss]

@homanchou please try using the version from the master branch.