Lax/traffic-accounting-nginx-module

Samples don't work with Elastic 7.x

kescherCode opened this issue · 3 comments

kescherCode commented

See title. The logstash's elasticsearch template will be rejected, having logstash throw some internal error.

kescherCode commented

Additional info: Elasticsearch and Logstash 7.10 currently don't want the template. I'm assuming it has something to do with the type fields, but other than that, I am not much further than I was on Feb 22.

[2021-05-30T00:29:18,602][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2021-05-30T00:29:18,613][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
...
[2021-05-30T00:19:37,573][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"version"=>1, "template"=>"ngx_accounting-*", "settings"=>{"index"=>{"refresh_interval"=>"5s"}}, "mappings"=>{"logs"=>{"_all"=>{"enabled"=>false, "norms"=>false}, "dynamic_templates"=>[{"accounting_status_fields"=>{"path_match"=>"@nr_status.*", "mapping"=>{"type"=>"integer"}}}, {"accounting_aggregate_status_fields"=>{"path_match"=>"@agg_status.*", "mapping"=>{"type"=>"integer"}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@from"=>{"type"=>"date"}, "@to"=>{"type"=>"date"}, "accounting_id"=>{"type"=>"keyword"}, "entry_type"=>{"type"=>"keyword"}, "nr_entries"=>{"type"=>"integer"}, "nr_open_entries"=>{"type"=>"integer"}, "nr_close_entries"=>{"type"=>"integer"}, "in_bytes"=>{"type"=>"integer"}, "out_bytes"=>{"type"=>"integer"}, "latency_ms"=>{"type"=>"integer"}, "upstream_latency_ms"=>{"type"=>"integer"}, "message"=>{"type"=>"text"}}}}, "aliases"=>{}}}
[2021-05-30T00:19:37,619][INFO ][logstash.outputs.elasticsearch][main] Installing elasticsearch template to _template/logstash
[2021-05-30T00:19:37,746][ERROR][logstash.outputs.elasticsearch][main] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://127.0.0.1:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:352:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:218:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.7.0-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in setup_after_successful_connection'"]}

Further though, Elasticsearch also reveals this:

[2021-05-30T00:32:51,271][DEPRECATION][o.e.d.a.b.BulkRequestParser] [unova.kescher.at] [types removal] Specifying types in bulk requests is deprecated.
kescherCode commented

I finally did some manual debugging to find out what error happens when logstash tries to install the template.

HTTP/1.1 400 Bad Request
Warning: 299 Elasticsearch-7.10.1-unknown "Deprecated field [template] used, replaced by [index_patterns]"
content-type: application/json; charset=UTF-8
content-length: 2255

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "Root mapping definition has unsupported parameters:  [logs : {dynamic_templates=[{accounting_status_fields={path_match=@nr_status.*, mapping={type=integer}}}, {accounting_aggregate_status_fields={path_match=@agg_status.*, mapping={type=integer}}}], _all={norms=false, enabled=false}, properties={accounting_id={type=keyword}, nr_open_entries={type=integer}, nr_close_entries={type=integer}, message={type=text}, nr_entries={type=integer}, latency_ms={type=integer}, in_bytes={type=integer}, @timestamp={type=date}, upstream_latency_ms={type=integer}, out_bytes={type=integer}, @to={type=date}, entry_type={type=keyword}, @from={type=date}}}]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters:  [logs : {dynamic_templates=[{accounting_status_fields={path_match=@nr_status.*, mapping={type=integer}}}, {accounting_aggregate_status_fields={path_match=@agg_status.*, mapping={type=integer}}}], _all={norms=false, enabled=false}, properties={accounting_id={type=keyword}, nr_open_entries={type=integer}, nr_close_entries={type=integer}, message={type=text}, nr_entries={type=integer}, latency_ms={type=integer}, in_bytes={type=integer}, @timestamp={type=date}, upstream_latency_ms={type=integer}, out_bytes={type=integer}, @to={type=date}, entry_type={type=keyword}, @from={type=date}}}]",
    "caused_by" : {
      "type" : "mapper_parsing_exception",
      "reason" : "Root mapping definition has unsupported parameters:  [logs : {dynamic_templates=[{accounting_status_fields={path_match=@nr_status.*, mapping={type=integer}}}, {accounting_aggregate_status_fields={path_match=@agg_status.*, mapping={type=integer}}}], _all={norms=false, enabled=false}, properties={accounting_id={type=keyword}, nr_open_entries={type=integer}, nr_close_entries={type=integer}, message={type=text}, nr_entries={type=integer}, latency_ms={type=integer}, in_bytes={type=integer}, @timestamp={type=date}, upstream_latency_ms={type=integer}, out_bytes={type=integer}, @to={type=date}, entry_type={type=keyword}, @from={type=date}}}]"
    }
  },
  "status" : 400
}