Receiving messages has side effects with offsets to commit
Closed this issue · 1 comments
It might be that I don't yet fully understand the usage of the kafka clients. When I was looking at
I had some spider senses tingling. For me the code read like
Whenever a message is completed we loop through OffsetsReceived https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L194 and commit all those that can be completed, essentially batching offsets to complete together but only triggered due to processing of messages.
Wouldn't it be more safe to do the following:
- Only complete things that can be truly associated with the message received
or apply smart background batching similar to what we do in Azure Service Bus.
The overall idea is to commit only messages that can be committed but without any hole in the sequence.
If we have launched offsets: 15,16,17,18 and we receive and ok for 17 we have to wait until 15 and 16 are processed. That's what is done there. This has to be done per topic/partition
I suppose this processing can be decoupled and done through a timer each second (configurable) or something like that.
How do you see this approach?