Kafka Queues?
darkfrog26 opened this issue · 2 comments
Still pretty new to Kafka, so perhaps I'm just missing something in the configuration, but I'm trying to create a horizontally scaling queue to allow many entries to be added to the queue and multiple servers to pick up messages from the queue without multiple servers receiving the same message. Can this be done with fs2-kafka? If so, can you please point me in the right direction?
Yes, Kafka supports this use case - as long as all the servers are in the same consumer group, each message will be delivered to only one server. https://blog.cloudera.com/scalability-of-kafka-messaging-using-consumer-groups/ has a good overview of this.
(Note - this isn't an actual guarantee - there are edge cases in which a message will be delivered more than once. But it holds to a sufficient degree to allow horizontal scaling as you describe.)
That article was very helpful, thank you. :)