Error with Painless scripted field 'doc['flow_id'].value'.

Question

Error with Painless scripted field 'doc['flow_id'].value'.

Opened this issue 4 years ago · 69 comments

Hi Ive tried to import the dashboards following the method

Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"runtime error","script_stack":["org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)","doc['flow_id'].value"," ^---- HERE"],"script":"doc['flow_id'].value","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2020.04.29-000001","node":"RmOnDn2mSsWSKkNKg2bgsA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)","doc['flow_id'].value"," ^---- HERE"],"script":"doc['flow_id'].value","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"No field found for [flow_id] in mapping with types []"}}}]},"status":400}

Im reading from a Remote PFSENSE via Filebeats. The logs hit Elastic after all of the filtering etc..

Thank you

Answer 1 · 2020-04-30T09:07:36.000Z

How do you import the dashboards exactly ?

Answer 2 · 2020-11-22T07:17:27.000Z

I'm receiving the same script exception. Dashboards, etc. are imported via the curl commands provided on the README page. The issue is preventing events in the EventsList from being displayed. I'm using the logstash filter that is linked to off the README page. The following is further information from the SN-ALL dashboard. Please advise.

script_exception at shard 0index logstash-flow-2020.11.22node VURsDiwmTnyNCTmjTmpqmQ
Type
script_exception
Reason
runtime error
Script stack
org.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:121)
org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:115)
'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase()
^---- HERE

Script
'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase()

Lang
painless
Position offset
73
Position start
0
Position end
232
Caused by type
illegal_state_exception
Caused by reason
A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!

Answer 3 · 2020-11-22T09:54:37.000Z

Was able to reproduce. Will try to cook a patch today. I think it is related to a possible fix here- StamusNetworks/SELKS#255 (comment)

I would like to confirm - on which dahsboars/vizs does this appear ?

Answer 4 · 2020-11-22T18:36:37.000Z

I only have Elasticsearch indexes for: alert, fileinfo, flow, http, tls. The issue is only appearing on SN-ALERTS from the data I have.

As a note, I attempted to use Filebeat to send Suricata logs directly to Elasticsearch using the elasticsearch7-template.json provided template. I verified the template was loaded in Elasticsearch. However, I believe my filebeat.yml file was incorrectly configured because I was only able to get a logstash- index, by modifying 'output.elasticsearch.index' and nothing was displayed in the dashboards. I'm not a Filebeat expert. If you have a filebeat.yml that works with the the template, it will eliminate the logstash service from the solution.

Answer 5 · 2020-11-24T13:11:21.000Z

Were the indexes created/existed in Kibana/Management ?

Answer 6 · 2020-11-24T17:56:01.000Z

The indexes were created through the logstash template provided off the README page. It is a slight modification given that 'type' doesn't exist in 7.x. The indexes did not exist prior to instantiating the stack.

Answer 7 · 2020-11-24T21:45:54.000Z

Ok - just to confirm , the issue appears only on SN-ALL or on SN-ALERTS, from the error it comes in from the logstash-flow... index which is not used i think in SN-ALERTS.

Answer 8 · 2020-11-24T22:11:37.000Z

I made a mistake in my last comment. It is only appearing on SN-ALL. I do not have any data in SN-ALERTS so I'm not able to confirm whether it occurs in SN-ALERTS.

Answer 9 · 2020-11-30T04:00:02.000Z

Any update on the above?

Answer 10 · 2020-11-30T07:32:57.000Z

This patch fixes the issue as mentioned here - #1 (comment)
It is either you can patch it up manually on each scripted field for each index - aka for example logstash-alert* / logstash-http* etc in Kibana Management .
Or it should also be taken care of on the next dashboards release, planned this week.
Apologies for the delay !

Answer 11 · 2020-11-30T22:26:45.000Z

No worries. Thank you for fixing. Fantastic work on these dashboards, btw!

Answer 12 · 2020-12-01T16:40:43.000Z

*Running SELKS 6 + ELK 7.10.0 + X-Pack enabled, so all communications are via https

I am having the same issue. So, the solution is just to enable the "community_id" in Suricata config and restart Suricata, or do I need to perform more steps?

Should I use doc['community_id.keyword'].value or doc['community_id'].value?

Thank you

Answer 13 · 2020-12-01T17:48:31.000Z

It does not seem the issue is related?
For enabling the community id - yest it just needs to be enabled and suricata restarted.

Answer 14 · 2020-12-01T18:39:31.000Z

Hi @pevma,

Like I said, I am experiencing the same issue. When I open Discover in Kibana, there's always a pop-up warning stating there is an issue with 2/15 shards. Please see the screenshots below:

This issue starts as soon I enable X-Pack and all the communications turned over https protocol. We have talked about this matter and some side effects this brings to SELKS suite in other posts. I was hopping that a new SELKS release or patch would fix this and other issues, that just appears if the user enables X-Pack with basic security features in ELK. Then I saw this post and I thought that maybe there is an easy way to address this issue, since other users have seen the same error.

I tried enabling the community_id in Suricata config, then restarted Suricata and Evebox. The issue do not disappear, just mutate into a different error, as you can see here:

It does not make any difference if I add or leave the .keyword. Maybe I am missing additional important steps.
I hope you can help me to make this error go away.

Thank you

Answer 15 · 2020-12-03T13:46:28.000Z

Any advise?

Answer 16 · 2020-12-03T22:04:56.000Z

Think you should use it without the .keyword
Before that you should make sure you see it properly in the json logs (eve.json) - there should be a community flow id key/record in the logs.

Answer 17 · 2020-12-04T14:06:36.000Z

Hi,

I only tried the .keyword because of this comment StamusNetworks/SELKS#255 (comment), but even that did not resolve the issue.

Checking the eve.json logs I can see flow_id field and also the community_id field:

{"timestamp":"2020-12-04T08:50:26.651146-0500","flow_id":1308048361440886,"in_iface":"enp2s0","event_type":"flow","src_ip":"192.168.1.128","src_port":58589,"dest_ip":"239.255.255.250","dest_port":3702,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":7,"pkts_toclient":0,"bytes_toserver":4886,"bytes_toclient":0,"start":"2020-12-04T08:47:26.378486-0500","end":"2020-12-04T08:47:33.171907-0500","age":7,"state":"new","reason":"unknown","alerted":false},"community_id":"1:JJD9J+CckkTq2iKzZP6j8zVZjNY="}
{"timestamp":"2020-12-04T08:50:26.651523-0500","flow_id":1308048361440886,"in_iface":"enp2s0","event_type":"flow","src_ip":"192.168.1.128","src_port":58589,"dest_ip":"239.255.255.250","dest_port":3702,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":7,"pkts_toclient":0,"bytes_toserver":4886,"bytes_toclient":0,"start":"2020-12-04T08:47:26.378486-0500","end":"2020-12-04T08:47:33.171907-0500","age":7,"state":"new","reason":"unknown","alerted":false},"community_id":"1:JJD9J+CckkTq2iKzZP6j8zVZjNY="}
{"timestamp":"2020-12-04T08:50:27.318169-0500","flow_id":2012176036617619,"in_iface":"enp2s0","event_type":"flow","src_ip":"192.168.1.179","src_port":50754,"dest_ip":"224.0.0.252","dest_port":5355,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":2,"pkts_toclient":0,"bytes_toserver":150,"bytes_toclient":0,"start":"2020-12-04T08:47:15.613779-0500","end":"2020-12-04T08:47:16.020953-0500","age":1,"state":"new","reason":"unknown","alerted":false},"community_id":"1:eR0XiX1AMxyOvQcJd8kGHF+YIzY="}
{"timestamp":"2020-12-04T08:50:27.318319-0500","flow_id":2012176036617619,"in_iface":"enp2s0","event_type":"flow","src_ip":"192.168.1.179","src_port":50754,"dest_ip":"224.0.0.252","dest_port":5355,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":2,"pkts_toclient":0,"bytes_toserver":150,"bytes_toclient":0,"start":"2020-12-04T08:47:15.613779-0500","end":"2020-12-04T08:47:16.020953-0500","age":1,"state":"new","reason":"unknown","alerted":false},"community_id":"1:eR0XiX1AMxyOvQcJd8kGHF+YIzY="}

The above logs are from a fresh SELKS 6 install and up to date, including ELK 7.10.0. I have not enabled the community_id field in suricata.yaml, but field is enabled in SELKS custom config file that overrides Suricata basic config (/etc/suricata/selks6-addin.yaml). So, the eve.json logs is including both fields: flow_id and community_id, and yet getting the shard errors related to the flow_id.

What would you recommend me to check/try next?

Thank you

Answer 18 · 2020-12-05T09:50:15.000Z

Where exactly are you making the change/addition in the scripted fields - is it in logstash-flow* index in Kibana management ?
And on what discovery/viz you exactly get the error ?

Answer 19 · 2020-12-07T14:05:40.000Z

Hi,

Error appears when I check app Discover/logstash-*. Error it is NOT present if I check Discover/logstash-flow-*. I tried modifications on Index Patterns/logstash-*. Index Patterns/logstash-flow-* does not have a scripted field.

Answer 20 · 2020-12-07T14:19:11.000Z

Ok - so you mean if you do discovery with the index logstahs-* ? What about if you try for example logstash-dns-* or logstash-http-*

Answer 21 · 2020-12-07T19:53:53.000Z

Verified one by one all logs in Discover/logstash-protocol-*. Only Discover/logstash-* it's being affected

Answer 22 · 2020-12-09T14:16:01.000Z

Any thoughts?

Answer 23 · 2020-12-09T15:46:57.000Z

What do you use the index logstash-service-* for ? Out of curiosity if ok to ask Apart from that I think it is a complain message - do the logs show up or not ?

…

-- Regards, Peter Manev On 9 Dec 2020, at 15:16, ManuelFFF ***@***.***> wrote: Any thoughts? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 24 · 2020-12-09T17:06:55.000Z

Hi,

I am sorry if I wasn't clear enough on my previous message, so you could be able to help me. Index logstash-service-* does not really exist. I tried to use a pattern name to refer to all the following indexes:

logstash-*
logstash-alert-*
logstash-anomaly-*
logstash-dhcp-*
logstash-dnp3-*
logstash-dns-*
logstash-fileinfo-*
logstash-flow-*
logstash-http-*
logstash-ikev2-*
logstash-krb5-*
logstash-nfs-*
logstash-rdp-*
logstash-rfb-*
logstash-sip-*
logstash-smb-*
logstash-smtp-*
logstash-snmp-*
logstash-ssh-*
logstash-tftp-*
logstash-tls-*

Perhaps I should have used logstash-[event_type]-* instead or just use the exact index name like this time. What I wanted to say is that I checked all the previous indexes, one by one, and the error comes only when I check Discover/logstash-*

Answer 25 · 2020-12-10T21:17:25.000Z

I think using logstash-event_type-* is better in terms of zooming in the specific index/event_type.
You can also look at any of the event types in their own dashboards including the raw events themselves at the bottom of every dashboard. So you just need to select the dashboard actually (From Kibana-> Dashboards) - for example SN-SMB will show you a dashboard with some visualizations and the raw logs of the event type SMB (or SMB protocol events).

Answer 26 · 2020-12-11T13:56:16.000Z

So, there is no way to fix this error?

Answer 27 · 2020-12-14T21:43:36.000Z

You should be able to import the raw API exports from here -
https://github.com/StamusNetworks/KTS7#how-to-use to fix the issue.

Answer 28 · 2020-12-23T06:28:04.000Z

Was this issue resolved in the master branch? I just pulled and I'm receiving the following:

script_exception at shard 0index logstash-flow-2020.12.23node n6KVwvteRyaKlBCWbQPACwTypescript_exceptionReasonruntime errorScript stackorg.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:121) org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:115) 'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase() ^---- HEREScript'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase()LangpainlessPosition offset73Position start0Position end232Caused by typeillegal_state_exceptionCaused by reasonA document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!

Answer 29 · 2020-12-23T07:15:20.000Z

Yes it is. Besides pulling the master branch you need to reload the dashboards The other alternative is simply to use the selks-upgrade_stamus routine - that will auto update the dashboards pkg.after which you can reset/reload it from the gui. https://github.com/StamusNetworks/SELKS/wiki/How-to-load-or-update-dashboards#from-scirius

…

-- Regards, Peter Manev On 23 Dec 2020, at 07:28, alphaDev23 ***@***.***> wrote: Was this issue resolved in the master branch? I just pulled and I'm receiving the following: script_exception at shard 0index logstash-flow-2020.12.23node n6KVwvteRyaKlBCWbQPACwTypescript_exceptionReasonruntime errorScript stackorg.elasticsearch.index.fielddata.ScriptDocValues$Longs.get(ScriptDocValues.java:121) org.elasticsearch.index.fielddata.ScriptDocValues$Longs.getValue(ScriptDocValues.java:115) 'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase() ^---- HEREScript'ip == ' + doc['src_ip.keyword'].value + ' && port == ' + doc['src_port'].value + ' && ip == ' + doc['dest_ip.keyword'].value + ' && port == ' + doc['dest_port'].value + ' && protocols == ' + doc['proto.keyword'].value.toLowerCase()LangpainlessPosition offset73Position start0Position end232Caused by typeillegal_state_exceptionCaused by reasonA document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 30 · 2020-12-23T19:55:20.000Z

I've recreated the entire ELK stack. Same issue. Please advise.

Answer 31 · 2020-12-23T20:34:34.000Z

I don’t think it should be needed to recreate the stack. What is the output of dpkg -l |grep stamus ?

…

-- Regards, Peter Manev On 23 Dec 2020, at 20:55, alphaDev23 ***@***.***> wrote: I've recreated the entire ELK stack. Same issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 32 · 2020-12-24T05:39:41.000Z

I'm using suricata 1:4.1.2-2 with filebeat 7.9.1 and ELK stack 7.9.1. The following is my filebeat config. The stack can easily be recreated if needed. I've done so just to ensure that the setup was from scratch. Thoughts?

filebeat.inputs:

input_type: log
enabled: true
paths:
- /var/log/suricata/eve.json

output.elasticsearch:
hosts: ["<ES_IP>:9200"]

Answer 33 · 2020-12-24T08:12:15.000Z

If you are not using the SELKS Kibana package you can import the ndjson directly - either via the Kibana management gui https://github.com/StamusNetworks/KTS7/tree/master/API-KIBANA7 or via the API - https://github.com/StamusNetworks/KTS7#how-to-use Please make sure you import the latest - in some cases if you are combining different visualisations in a custom dashboard you might need to remove and then import the same viz into the dashboard in order fir the changes to be picked up(I’ve experienced that a couple of times with custom created dashboards)

…

-- Regards, Peter Manev On 24 Dec 2020, at 06:39, alphaDev23 ***@***.***> wrote: I'm using suricata 1:4.1.2-2 with filebeat 7.9.1 and ELK stack 7.9.1. The following is my filebeat config. The stack can easily be recreated if needed. I've done so just to ensure that the setup was from scratch. Thoughts? filebeat.inputs: input_type: log enabled: true paths: /var/log/suricata/eve.json output.elasticsearch: hosts: ["<ES_IP>:9200"] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 34 · 2020-12-24T17:06:20.000Z

I'm using the following commands to load the objects (from the API-KIBANA7) directory. This is the same thing that you mention above, correct? These were executed after a cloning anew as of my comment 2 days ago.

curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form file=@index-pattern.ndjson
curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form file=@search.ndjson
curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form file=@visualization.ndjson
curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form file=@dashboard.ndjson
curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form file=@query.ndjson**

Answer 35 · 2020-12-24T22:08:59.000Z

Yep , that should do it. Can you please share the output of those commands?

…

-- Regards, Peter Manev On 24 Dec 2020, at 18:06, alphaDev23 ***@***.***> wrote: I'm using the following commands to load the objects (from the API-KIBANA7) directory. This is the same thing that you mention above, correct? These were executed after a cloning anew as of my comment 2 days ago. curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form ***@***.*** curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form ***@***.*** curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form ***@***.*** curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form ***@***.*** curl -X POST "suricata_kibana:5601/api/saved_objects/_import" -H 'kbn-xsrf: true' --form ***@***.***** — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 36 · 2020-12-24T23:52:27.000Z