ankane/pdscan

Idea: Add pagination in Elasticsearch

Opened this issue · 0 comments

If you set the sample to more than 10,000 (elasticsearch default) it will crash.

Error: [400 Bad Request] search_phase_execution_exception: all shards failed

The reccomended way to do deep pagination is using search_after
https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after

I'm not familiar with go , but that functionality is currently available in the Go client:
https://github.com/elastic/go-elasticsearch/blob/4ea9db9cf63079fd1dbdc92e44737869f40589d9/_examples/xkcdsearch/store_test.go#L130