New partitions are not seen by existing consumers
bataras opened this issue · 7 comments
Using these tools:
- saram-cluter-cli
- kafka manager UI
- a simple producer writing every second to a topic, logging which topic/partition it went to
Create a single, 1 partition topic.
Run the producer. Then run the CLI tool like this..
./sarama-cluster-cli -brokers <a broker addr) -group test -topics <a topic> -offset oldest
The cli tool happily consumes messages. But if you go to the Kafka Manager and add a partition to it, the Producer code sees a rebalance and starts writing alternately to both partitions, but the CLI consumer does not see the new partition and thus starts missing half the messages.
If you stop and rerun the CLI, it sees both partitions and catches up.
Problem is the sarama-cluster consumer needs to be able to automatically pick up new partitions
This issue is about a consumer not picking up new partitions on an existing topic, while a produce does. It's not about picking up new topics
My bad, I misread your post, and assumed it was related to the problem I was running into. I'll remove it to avoid confusion. 👍
@Preylien please see my comment on #170 (comment).
@dim I don't think this issue is closed. I misread the problem bataras was having, so I believe it should be reopened and looked into separately. Sorry about any confusion.
Right. This issue is about a live consumer group not recognizing when a new partition is added to a topic
May I know if any updates on this issue?
@FelixChong no updates, sarama-cluster is now officially deprecated since IBM/sarama#1099 was merged and released. The native sarama implementation makes it easy to watch topics and change subscriptions on rebalances.