Bulk upload a custom store / Store with params
silviurosu opened this issue · 4 comments
I have a use case were I need to reindex a big part of the documents but not all. For example all restaurants from a channel. Since they can be thousands it's very slow to send documents one by one.
I would need to use bulk upload but being able to send a custom stream built on runtime.
There is no way at the moment to send params to the store when building the stream.
Can we add the option to send params to Bulk.upload to be sent to the store?
Or to expose at least a put_bulk method in Elasticsearch module that receives a list of items and perform a bulk upload?
Yeah, that's a good suggestion. Overall, I would like to make the API of the package rely less on stores and more on plain old functions.
Can you give me more hints on how you want to tackle this issue? I want to be sure before digging in the code.
Should I add another method put_bulk?
Or should I try to inject the stream in existing methods?
Or in case you want to more away from streams. then maybe we should suffice having a put_bulk that receives a list of documents and it up to user how it loads those documents?
Hi @silviurosu, I'm currently working on a partial rewrite of the library that will maintain most of the current API, but expose the lower-level functions for direct bulk upload, etc. It should also eliminate the need for a store. Timeframe is hard to estimate right now, but I am working on it most weeks.
Ok, thanks for the notice. So I will not work on this and wait for your changes.
Let me know if I can help with something.