Page through the results in `data.elasticsearch` data source
traut opened this issue · 0 comments
traut commented
Description
Elasticsearch caps the result window for a search request at 10K results. If the size is set to more than 10K, the error is returned:
[400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way
to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards
failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".internal.alerts-security.alerts-default-000001","node":"YEi3-SiNQ4-2ZM1tVOYKAg","reason":{"type":"illegal_argument_exception","reason":"Result window is too large, from
+ size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level
setting."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way to request large data sets. This
limit can be set by changing the [index.max_result_window] index level setting.","caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the
scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}},"status":400}
Environment
Fabric version: 0.4.1
elastic plugin version: 0.4.1
Steps to Reproduce
The error can be reproduced by settings size value to a number higher than 10000.
Expected Behavior
Fabric should support any non-negative int value in size attribute, with the limiting factor being a gRPC call message size (20mb).