Clarification NoOffsetReset
esiqveland opened this issue · 3 comments
I am having some trouble understanding exactly how NoResetOffset() works.
I am working only with group consumers. My use case is that I want to set the default behaviour to not touch any offset if there is are no offset, and ideally error on startup to let the caller know they have to make a choice what to do.
Reading the docs I think NoResetOffset() gives a little conflicting information:
Line 69 in 351e7fa
// NoResetOffset returns an offset that can be used as a "none" option for the
// ConsumeResetOffset option. By default, NoResetOffset starts consuming from
// the beginning of partitions (similar to NewOffset().AtStart()). This can be
// changed with AtEnd, Relative, etc.
//
So if I pass kgo.ConsumeResetOffset(kgo.NoResetOffset()) and the consumerGroup does not exist from earlier, will it refuse to start or start conusming from beginning, ie. same as NewOffset().AtStart() ? In the second case, how can I stop that from happening and ideally signal the caller that they need to actively set a ResetOffset policy?
Sorry, ConsumeResetOffset is a really confusion option that I poorly named when tagging 1.0 and now the library is stuck with it.
NoResetOffset is meant to be a modifier after OffsetOutOfRange is encountered. It doesn't affect how you start consuming.
Looking at the code, the requested timestamp is initialized to MinInt64, but with the way Kafka works, that's functionally equivalent to requesting with the special timestamp -2 ("give me the start offset"). If the requested timestamp is not -1 nor -2, then Kafka returns the first offset after the requested timestamp. So, if the requested timestamp is min int, Kafka will return the first offset.
I think you want a separate new behavior, NoInitialOffset (or some good name). I'll see if this is easy and if so, squeeze it in the next release. I don't think this would be good to bundle with NoResetOffset, because that's specifically around resetting in the face of a single specific error.
Hello, thank you the quick response. I am not very familiar with the details of OffsetOutOfRange. It seems like this is for situations where you have/found an offset, and when fetching this offset, the offset is before the earliest offset, ie fetchOffset < firstOffset.
I realize my intent might not be perfectly clear.
Essentially I am looking to set the equivalent of auto.offset.reset=none (See: https://docs.confluent.io/platform/7.6/installation/configuration/consumer-configs.html#auto-offset-reset)