Possibility to tap into already executed data
stychu opened this issue · 0 comments
stychu commented
Hi I want to hit my external API with bunch of requests. API allows for 30 TPS. I need to execute 500 requests which each request returns 100 objects that I need to save to the database. Is it possible to tap into the queue while the queue is still running and access already resolved data? It will be a bit heavy process on memory where I need to wait till the whole queue resolve and only then I can access my in memory result of 50 000 objects to be saved into database.
If I have pqueue as
const queue = new PQueue({
intervalCap: 30,
interval: 1000,
carryoverConcurrencyCount: true,
});
const result = await queue.addAll([500requestPromises])
I would like to be able to tap into the data and every batch save to database so approx. 17 writes to database. Is it possible ??