logstash-plugins/logstash-input-elasticsearch

Allow the use of the size parameter inside a query

Opened this issue · 0 comments

For all general issues, please provide the following details for fast resolution:

  • Version: 7.9.1
  • Config File (if you have sensitive info, please remove it):
input {
  elasticsearch {
    cloud_id => "<deployment_endpoint>"
    cloud_auth => "<username>:<password>"
    index => "<filebeat-*>"
    schedule => "*/5 * * * *"
    query => '{ "size": 10, "query": { "bool": { "must_not": [ { "query_string": { "query": "(starting)", "default_field": "status" } } ], "filter": [ { "bool": { "must_not": { "exists": { "field": "event_ts" } } } } ] } } }'
    docinfo => true
  }
}
  • Use case: Replacement for the Elapsed filter plugin ingesting Kafka logs
  • Sample Data: Kafka logs
  • Workaround: Reducing the number of hits with a range filter or by limiting the number of indices to query...
  • Steps to Reproduce:

While using the Elasticsearch input plugin to query against a set of indices, it seems not possible to limit the number of documents returned by the query.

The size parameter of the Elasticsearch input plugin is used with scroll queries to limit the number of items per page in the scroll, not the total number of documents.

And the size parameter inside the query is ignored and the query always returns the default 10.000 documents.

This was reported in the discuss forum with a simple example:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

I'd like to be able to use the size parameter inside the query field to limit the number of documents returned.

Related: #85