[Consumer] Consume at the latest rather than the earliest offset
rgraber opened this issue · 1 comments
rgraber commented
The consumer is currently set to 'auto.offset.reset': 'earliest', meaning it will always go to the earliest unprocessed offset in that consumer group. This means that if an event has been publishing for a while before the consumer is deployed, the consumer will try to process all previous events in the topic, which could be a significant backlog. We should disable this behavior. Just removing the auto.offset.reset line should do it, since consumers default to latest.
A/C
- New consumers begin processing at the latest unprocessed offset in the group.
robrap commented
- Reminder do update the docs here: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus#Special-Kafka-Configurations
- Can we mark this as done? I don't think this needs to be tested in Prod, if tested elsewhere.