Allow to disable event-based indexation
mvsmal opened this issue · 8 comments
In the latest version, the event-based indexation has been introduced.
There should be a way to disable it. Either allow to unsubscribe (then the handler must be a publicly available method, but not a lambda and the registrar should a corresponding method for unsubscribing), or provide a setting, allowing to disable such behavior. The reason for such request is that we are importing thousands of products at once and we get a huge amount of enqueued jobs in Hangfire. Overall it has huge impact on the performance.
Version info:
- Browser version: any
- Platform version: 2.13.50
- Module version: 2.23.27
There is a way to disable event-based indexation in code (used in platform import):
using (var guard = EventSupressor.SupressEvents())
{
// Import data
…
}
But we just use POST /api/catalog/products
Ok, we will think how to handle this.
After internal discussion we need more details on the problem.
We generate one job for every POST /api/catalog/products batch. How many products are included in one batch? How many products in all batches?
How do you observe performance decreasing? I mean let's say we disable event-based indexation, then you loaded 10k products at once. Then you run the indexation, and it should handle all these products. Maybe it is not better for performance to have 1 task with 10k products than 100 tasks with 100 products. Or the problem is that many hangfire tasks consume all processors, while single does not?
POST /api/catalog/products expects just one product, so there is no batch. We could have thousands of such requests at once. Having the same environment, the whole re-indexation of products takes ~1 hour, but event based indexation of the same amount of products takes ~4 hours.
I see the difference, and it is a problem, indeed. We should have "protection" for such scenarios (e.g. throttling).
As optimization, could you try to use POST /api/catalog/products/batch for the import? Seems it allows to save products by batches. I think reducing indexing tasks count could significantly decrease total time.
@t13ka Could you please check if turning on Indexation job by schedule turn off indexation by events?