taskforcesh/bullmq-pro-support

Is there any possible make group and batch work together?

Opened this issue · 3 comments

0xuhe commented
// worker.ts
const worker = new WorkerPro(
  "aaa",
  async (job: JobPro) => {
    console.log("start");
    // await sleep(12e3);
    const jobs = job.getBatch();
    for (const job of jobs) {
      console.log(job.data);
    }
    console.log("end");
  },
  {
    connection: redis,
    concurrency: 20,
    group: {
      concurrency: 1, // Limit to max 3 parallel jobs per group
    },
    batch: { size: 10 },
  }
);
const queue = createQueue("aaa");

await queue.add(
    "bbb",
    { a: 1 },
    {
      group: {
        id: "aaa",
      },
    }
  );
await queue.add(
    "bbb",
    { a: 2 },
    {
      group: {
        id: "aaa",
      },
    }
  );

desire:

start
{a: 1}
{a: 2}
end

actually

start 
{a:1}
end
start
{a:2}
end
manast commented

Currently, this is not supported. Could you please tell us a bit more about your use case and why this is important for you?

0xuhe commented

@manast

In my case, I need jobs to change host's IPs, a host only change one IP or many IPs in one process, but could not process meanwhile on the same host, each process cost 30s.
First, there are many workers to process corresponding hosts at same time.
Second, When a worker receive a job, the worker will process its job in 30s, the next jobs will wating for this job completed. There maybe many jobs waiting for first job to be completed. After first job completed, the worker could process wating jobs in batch.
eg:
I have many hosts and IPs need change.
hostA: Ip1
hostA: Ip2
hostA: Ip3
hostB: Ip1
hostB: Ip2
hostB: Ip3

My desire:
queueA: hostA
queueB: hostB
works in the meantime. It will process Ip1 immediately and it will cost 30s, and this queue receive job Ip2 at 5s and job Ip3 at 10s. I wish the worker could process Ip2 and Ip3 in batch after job Ip1 complete at 30s.

manast commented

Thanks. We will make an analysis to understand how feasible it is to add batch support to groups.