beatlabs/patron

Kafka consumer consolidation

mantzas opened this issue · 0 comments

Is your feature request related to a problem? Please describe

Up to this point, we had two consumer implementations, simple and group.
Group used consumer groups to handle the messages whereas simple consumed from each partition independently.
The reason for simple was to be able to handle messages with better support for offset since we could be handling the messages for each partition independently.

The same can be achieved by setting the offset to the latest and using as consumer group always a different unique value per pod e.g. a UUID. Kafka has a garbage collector for unused consumer groups following this value.

Describe the solution

We should be moving forward and using only group one and making it the only one available.