BulkIngester should allow customizations of intermediate requests
fabriziofortino opened this issue · 0 comments
Description
Using the HLRC BulkProcessor, it became feasible to intercept the creation of requests by utilizing a BiConsumer that was invoked every time a bulk request was constructed (refer to https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/bulk/BulkProcessor.java#L213).
This capability proved particularly advantageous in various situations. For instance, if we consider a scenario where there are 1000 operations divided into 10 bulk requests, I could intercept the final request and modify the request refresh policy (for example, request.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);).
Having a similar feature in the BulkIngester would be highly beneficial. One straightforward approach could involve adjusting the visibility of BulkIngester.newRequest() (refer to https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/elasticsearch/_helpers/bulk/BulkIngester.java#L243)