Uneven Partition Leg
laukik85 opened this issue · 5 comments
Describe the bug
I am facing issue where lag is accumulated in one or a small number of partitions, then the backlog also falls on one or a small number of consumers.
Is there anyway I can handle this situation?
Hello @laukik85 !
It would be nice to know what caused this lag. There can be many reasons.
For example if that broker where the problematic partition gets a big chunk of data and the consumer could not process it in time.
I would suggest using more partitions for the topic and using more consumer with consumer group for parallelizing.
You can check also the partition distribution in the Cruise Control UI. When self healing is used in Cruise Control partition distribution happens automatically (
@laukik85 It'd be appreciated if you can provide some updates on this so we can proceed
Hi @panyuenlau,
Sorry for delayed posting on this.
As suggested by @bartam1, we worked on consumer side to improve performance (processing/database injection) and it really helped healing single partition leg problem.
I really appreciate your help on this.
Hi @panyuenlau,
Sorry for delayed posting on this.
As suggested by @bartam1, we worked on consumer side to improve performance (processing/database injection) and it really helped healing single partition leg problem.
I really appreciate your help on this.
@laukik85 Thanks for the update and glad to hear that you can resolve the issues with suggestions from our team member. With that said, it seems like we can close this issue because there isn't anything that we need to do on the koperator, right?
Hi @panyuenlau,
Sorry for delayed posting on this.
As suggested by @bartam1, we worked on consumer side to improve performance (processing/database injection) and it really helped healing single partition leg problem.
I really appreciate your help on this.
@laukik85 Thanks for the update and glad to hear that you can resolve the issues with suggestions from our team member. With that said, it seems like we can close this issue because there isn't anything that we need to do on the koperator, right?
Yes, that's correct.