robcowart/synesis_lite_snort

Index to elasticsearch not possible

rkscon opened this issue · 5 comments

Hello,
i encounter the following issue out of the logstash log:

[2019-01-18T11:53:18,482][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"snort-1.0.0-2019.01.18", :_type=>"doc", :routing=>nil}, #LogStash::Event:0x7292b14b], :response=>{"index"=>{"_index"=>"snort-1.0.0-2019.01 .18", "_type"=>"doc", "_id"=>"UYXOYGgB9Zyq9vh65aGM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [event.host] of type [keyword]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:77"}}}}}

I installed the ELK Stack fresh.
Filebeat is shipping logs from snort maschine to the server where the ELK stack is running. Further i see the Index in Index Management of Elasticsearch .

As far as i understand, there must be some kind of problem to correctly index the Data.

I am quite new to this topic and not aware of any possiblity to debug this any further

Thank you all in advance
Greetings
Simon
P.S.: Sry for posting in the wrong project.

I have same issue on logstash-plain.log

when I flow the below link

https://github.com/robcowart/synesis_lite_snort

2019-02-19T17:18:15,169][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"snort-1.0.0-2019.02.19", :_type=>"logs", :_routing=>nil}, 2019-02-19T06:12:57.313Z %{host} %{message}], :response=>{"index"=>{"_index"=>"snort-1.0.0-2019.02.19", "_type"=>"logs", "_id"=>"t_NnBGkBJ7r60F19pLY4", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [event.host] of type [keyword]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:508"}}}}}

Same problem! =/

If you are using elastic 7.x the index will not work. You have to use 6.x or change the template.
I have no working template yet but using default mapping is not allowed anymore.

Loading the current template in 7.x will give you the Could not index event to Elasticsearch. {:status=>400, :action= error

https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

"mappings": {
"default": {
"numeric_detection": true,
"dynamic_templates": [
{
"string_fields": {
"match_mapping_type": "string",
"match": "*",
"mapping": {
"type": "keyword"
}
}
}
],

Elasticsearch 7.x

    Specifying types in requests is deprecated. For instance, indexing a document no longer requires a document type. The new index APIs are PUT {index}/_doc/{id} in case of explicit ids and POST {index}/_doc for auto-generated ids. Note that in 7.0, _doc is a permanent part of the path, and represents the endpoint name rather than the document type.
    The include_type_name parameter in the index creation, index template, and mapping APIs will default to false. Setting the parameter at all will result in a deprecation warning.
    **The _default_ mapping type is removed.**

Closing all issues as this project has been archived.