Consumer batch processing

Question

Consumer batch processing

shaiyahleba opened this issue 4 years ago · 9 comments

Hi,

Is it possible to process a batch of records in a single consumer instead of just a single record at a time?

To achieve greater parallelism beyond the number of the topic's partition, it would be great if we could process in parallel (or even concurrently) a batch of records on the consumer side.

Thanks

Answer 1 · 2021-03-26T13:35:18.000Z

Hi @shaiyahleba ,
I'm glad to report we are working on a batch consumer feature, and we will open-source it soon.
I will update this issue once this happens.

Natan

Answer 2 · 2021-03-26T13:36:38.000Z

btw, Are you currently using Greyhound.
Can you share which API? ZIO/Scala Future/Java?

Answer 3 · 2021-03-26T17:08:57.000Z

Great thanks! Any ETA?

We’re using the Java api.

Answer 4 · 2021-03-27T19:30:58.000Z

Do you think you can contribute to the Java api of batch consumer?
It could make the feature available faster.

Answer 5 · 2021-03-29T12:56:53.000Z

Unfortunately not at the moment.

Answer 6 · 2021-03-31T09:30:19.000Z

I am trying to make this task a priority in upcoming quarter. I hope to update here soon on ETA

Answer 7 · 2021-03-31T19:16:27.000Z

Great thanks

Answer 8 · 2021-04-08T17:17:39.000Z

@shaiyahleba
The plan is to have batch consumer available by end of May.
FYI

Answer 9 · 2021-05-31T10:03:55.000Z

@shaiyahleba batch consumer is ready! (it's found in version 0.2.0)