Missing current-offset and lag when describing connector's consumer groups

Question

Missing current-offset and lag when describing connector's consumer groups

avelanarius opened this issue 3 years ago · 5 comments

A problem reported by a user on Slack. When using kafka-consumer-group.sh to describe the consumer group of the Sink Connector, some of its details (current-offset and lag) are not correctly filled in:

Answer 1 · 2022-03-10T17:14:37.000Z

Issue successfully reproduced locally by me. I can confirm (as the user report stated) that the problem is present only when storing offsets on Scylla is disabled.

Answer 2 · 2022-03-14T11:51:24.000Z

The only difference that could explain why it works when storing offsets on Scylla is enabled, are the code blocks guarded by if (config.isOffsetEnabledInScyllaDb()). There are 6 blocks like this, but only 2 look suspicious (both in ScyllaDbSinkTask). One that is probable to be the culprit is the one in preCommit. Maybe changing this one to call super.preCommit(...) on non-Scylla offset storage would fix the issue - to be further investigated.

Answer 3 · 2022-03-18T14:53:20.000Z

Issue successfully reproduced locally by me.

Unfortunately, this was a premature celebration. Upon actually inserting some data into the replicated topic, the columns are correctly updated. @Bouncheck has similar problems to reproduce the issue.

Answer 4 · 2022-04-04T12:34:31.000Z

any idea/guidance on how to monitor for example the lag with this connector?

Answer 5 · 2022-04-20T10:21:59.000Z

Upon further inspection, using kafka-consumer-group.sh is a good way to monitor the lag of the connector. We were unable to reproduce any issues with it and monitoring with kafka-consumer-group.sh works in both modes of storing offsets (both on Kafka and on Scylla).