Duplicate messages received
jransome opened this issue · 1 comments
We are using a v5.0.0 ConsumerGroup (node 12) to consume messages and manually commit them once we have finished processing them. On each message, we call pause() on the consumer, process the message (which can take on average 700ms +/-200ms), call commit() and then resume() the consumer. However this results in the occasional receipt of duplicate messages under certain conditions where there is a backlog (at least >100 messages) where the consumer will emit x number of messages between commits followed by a repetition of the last n number of messages in that batch of x. We are using the following configuration that may be relevant:
fetchMaxBytes: 524288
fetchMinBytes: 10 * 1024
maxTickMessages: 50 (also up to 100)
autoCommitIntervalMs: 1000
autoCommit: false
Is there something that might explain this repetition of the last n number of messages between commits?