Exception :exception=>#<TypeError: can't convert Java::JavaUtil::ArrayList into Hash>
fedelemantuano opened this issue · 8 comments
Hi,
I can't parse a valid json with Logstash. I have the follow error:
Trouble parsing json {:source=>"message", :raw=>"[ [ { \"key1\": \"value1\" } ], [ { \"key2\": \"value2\" } ]]", :exception=>#<TypeError: can't convert Java::JavaUtil::ArrayList into Hash>, :level=>:warn}
I'm trying to parse a complex valid json file, but I reproduced the error with the follow json:
fedelemantuano@solaria ~/Downloads $ cat test.json
[
[ { "key1": "value1" } ],
[ { "key2": "value2" } ]
]
fedelemantuano@solaria ~/Downloads $ jsonlint test.json
test.json: ok
I can parse with python:
In [8]: import json
In [9]: test = json.loads('[ [ { "key1": "value1" } ], [ { "key2": "value2" } ] ]')
In [10]: test
Out[10]: [[{u'key1': u'value1'}], [{u'key2': u'value2'}]]
but with Logstash I have the follow error:
Trouble parsing json {:source=>"message", :raw=>"[ [ { \"key1\": \"value1\" } ], [ { \"key2\": \"value2\" } ]]", :exception=>#<TypeError: can't convert Java::JavaUtil::ArrayList into Hash>, :level=>:warn}
{
"message" => "[ [ { \"key1\": \"value1\" } ], [ { \"key2\": \"value2\" } ]]",
"@version" => "1",
"@timestamp" => "2015-10-07T16:30:22.326Z",
"headers" => {
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "POST",
"request_path" => "/",
"request_uri" => "/",
"http_version" => "HTTP/1.1",
"http_user_agent" => "curl/7.38.0",
"http_host" => "koro.certego.local:8080",
"http_accept" => "*/*",
"content_length" => "59"
},
"tags" => [
[0] "_jsonparsefailure"
]
}
Can we help me, please?
Buongiorno @fedelemantuano
This doesn't look like a valid json, i understand that jsonlint is saying that it is, but i think that this json should look like:
{
"jsoninput": [
{
"key1": "value1"
},
{
"key2": "value2"
}
]
}
Then, the output from logstash will be something like:
{
"message" => "{\"jsoninput\":[{\"key1\":\"value1\"},{\"key2\":\"value2\"}]}",
"@version" => "1",
"@timestamp" => "2015-10-08T13:02:54.011Z",
"host" => "Gabriels-MacBook-Pro.local",
"jsoninput" => [
[0] {
"key1" => "value1"
},
[1] {
"key2" => "value2"
}
]
}
Hi @gmoskovicz thanks for the answer.
I'm using the JsonML library (http://www.jsonml.org/) to convert HTML in JSON and the output JSON is like this:
[
"ul",
[
"li",
{
"style": "color:red"
},
"First Item"
],
[
"li",
{
"title": "Some hover text.",
"style": "color:green"
},
"Second Item"
],
[
"li",
[
"span",
{
"class": "code-example-third"
},
"Third"
],
" Item"
]
]
I used the follow validators:
and for them is a valid JSON. I can parsing with python.
It's possible itegrate Logstash JSON plugin to manage this JSON?
@fedelemantuano and @gmoskovicz
Firstly, the input is correct JSON.
Secondly, this is a bug. https://github.com/logstash-plugins/logstash-filter-json/blob/master/lib/logstash/filters/json.rb#L70 does not test for ArrayList as well.
Thirdly, this will be automatically fixed in LS v2.0 (due in a few weeks).
Fourthly, I will fix this bug in a gem version that will work with LS 1.5.4
Hi,
I have the same issue with logstash 2.1.1. I tested the same json and I had another error message:
Trouble parsing json {:source=>"message", :raw=>"[ \"ul\", [ \"li\", { \"style\": \"color:red\" }, \"First Item\" ], [ \"li\", { \"title\": \"Some hover
text.\", \"style\": \"color:green\" }, \"Second Item\" ], [ \"li\", [ \"span\", { \"class\": \"code-example-third\" }, \"Th
ird\" ], \" Item\" ]]", :exception=>#<RuntimeError: Parsed JSON arrays must have a destination in the configuration>, :level=>:warn}
Maybe there is the same bug.
@fedelemantuano Looks like this bug was not fixed. @guyboertje can you confirm this?
Yes @gmoskovicz the bug is open, but in last answer @guyboertje told me:
Thirdly, this will be automatically fixed in LS v2.0 (due in a few weeks).
Fourthly, I will fix this bug in a gem version that will work with LS 1.5.4
and the error message is changed from TypeError: can't convert Java::JavaUtil::ArrayList into Hash to exception=>#<RuntimeError: Parsed JSON arrays must have a destination in the configuration> with the same json.
@fedelemantuano, @gmoskovicz - the root cause is the same. The problem shows itself differently.
I understand the full picture now.
You need to specify a target field config option. See the docs
This is why @gmoskovicz suggested that it is not valid JSON - I think Gabriel meant invalid in the LS context.
If your config is:
filter {
json {
# Parse message as JSON
source => "message"
}
}
For an Event like this:
"message" => "{\"hello\" : \"world\"}"
"other" => "value"
Note that the source field is a JSON object. In this case, a target is not needed because the JSON objects keys and values are transferred to the Event so you get this:
"message" => "{\"hello\" : \"world\"}"
"other" => "value"
"hello" => "world"
If, as in @fedelemantuano case the source field is not a JSON object then a target field must be set in the codec config. Imagine your config is:
filter {
json {
# Parse message as JSON, store the results in the 'html' field
source => "message"
target => "html"
}
}
For this Event:
"message" => "[ \"li\", { \"style\": \"color:red\" }, \"First Item\" ]"
"other" => "value"
you get:
"message" => "{\"hello\" : \"world\"}"
"other" => "value"
"html" => [ "li", { "style" => "color:red" }, "First Item" ]
Note that field "html" now contains Ruby objects parsed from your JSON array.
So in closing, the original error "TypeError: can't convert Java::JavaUtil::ArrayList into Hash" was fixed.
@fedelemantuano - can you close this now?
Yes,
I tested without elasticsearch output plugin, and the json filter work.
Now I have a issue in elasticsearch:
MapperParsingException[object mapping for [message_parsed] tried to parse field [null] as object, but found a concrete value]
But also for me this issue is solved.
Thank you so much.