logstash-plugins/logstash-output-elasticsearch

Reduce ES response size through use of filter_path

jsvd opened this issue · 3 comments

jsvd commented

see elastic/beats#36275

during the _bulk request if "errors,items.*.error,items.*.status" is applied to filter_path, it should reduce > 90% of the response size from ES, reducing bandwitdh usage and also making response processing faster (less json to deserialize).

Cool, can we track performance implications on Logstash instances as this gets rolled out?

jsvd commented

For those wanting to test it out, current versions of Logstash allow configuring the bulk path, where the filter_path can be set:

output {
  elasticsearch {
    hosts => [ ... ]
    bulk_path => '_bulk?filter_path=errors,items.*.error,items.*.status'
  }
}

I did a quick initial benchmark with 2GB of data in a simple file input/elasticsearch output pipeline and I see a roughly 5% drop in CPU usage and a 75% drop in response traffic from Elasticsearch.