Reduce ES response size through use of filter_path
jsvd opened this issue · 3 comments
jsvd commented
during the _bulk request if "errors,items.*.error,items.*.status"
is applied to filter_path, it should reduce > 90% of the response size from ES, reducing bandwitdh usage and also making response processing faster (less json to deserialize).
flexitrev commented
Cool, can we track performance implications on Logstash instances as this gets rolled out?
jsvd commented
For those wanting to test it out, current versions of Logstash allow configuring the bulk path, where the filter_path
can be set:
output {
elasticsearch {
hosts => [ ... ]
bulk_path => '_bulk?filter_path=errors,items.*.error,items.*.status'
}
}
strawgate commented
I did a quick initial benchmark with 2GB of data in a simple file input/elasticsearch output pipeline and I see a roughly 5% drop in CPU usage and a 75% drop in response traffic from Elasticsearch.