bsm/sarama-cluster

how to limit number of partitions assigned to consumer

nitin302 opened this issue · 2 comments

nitin302 commented

For Kafka, partitions provide a way for consumption to scale. Hence, it is necessary that consumers needs to scale horizontally as well.

  1. Is there any way to limit number of partitions per consumer, so that we can add more consumers so that all partitions are assigned to one of the consumers (total consumers * partitions per consumer = total partitions of the topic)? How can I do this?

  2. What should the configuration of the node in this case? What logic can be used for size a consumer for a given number of partitions?

dim commented

@nitin302 I suggest that you look into Shopify/sarama#1099 which has been merged and released, as sarama-cluster is now deprecated. You can configure a custom https://godoc.org/github.com/Shopify/sarama#BalanceStrategy which can plan the partition allocation to consumers based on a consumer's UserData. Hope that helps!

Shan4Kites commented

@dim can you please add "deprecated" in the github description for this repo? it will help the new users.