noamt/elasticsearch-grails-plugin

DoBulkRequest in descending id order

Opened this issue · 0 comments

Dear maintainers,
Today I was discussing with my team a feature that I feel it could be very nice for you to incorporate. We added this to our JIRA board to extend our current version of the elasticsearch plugin locally to add this.

The feature is: being able to define if the bulk index queue should be filled with instances considering an ascending (default) or descending id order.

Why can this be useful?
Consider that the indexing can take a long time to complete (ours take 4 hours because of reasons). In our case, instances with lower IDs are also older instances, which are less relevant than newer instances. As an instance gets indexed, its document is ready to be searched. In my system is more interested to have newer instances ready for search than older instances, thus the descending order makes more sense to us. Maybe other users have a similar use case.

How to do it?
Our local 5-minute change will be quite simple, we just need to change 'asc' to 'desc' in one place, but someone that wants to take this request in can make it fancier if they feel like it.
At ElasticSearchService the method doBulkRequest has the following code:

def results = scm.domainClass.clazz.withCriteria {
    firstResult(resultToStartFrom)
    maxResults(maxRes)
    order('id', 'asc')
}

My fancier suggestion is to allow to set the preferred indexing order somehow, maybe expose a way to alter the SearchableClassMapping, add a parameter there to hold "asc" or "desc" in case you feel it's important to let the indexing order to be set per Domain. In this case, we would also need a method to alter all SearchableClassMapping at once.
Otherwise, my zero fanciness suggestion is to add to the index call a "Map params = [:]" where one could just pass the [order: "desc"] if wanted and it would be done.