consumer group offsets by partition missing

[mdkouki]

Hi,
First of all a big Thank you for the work you are doing in this project, with kafka minion we used to have this metric "kafka_minion_group_topic_partition_offset" that was useful to see the group position in a specific topic - partition log , now kminion_kafka_consumer_group_offset_commits_total (a scope of the consumer group with all topics) and kminion_kafka_consumer_group_topic_offset_sum ( topic-consumer group scope) give I larger scope idea about the offsets but since the kafka unit scope is the partition we miss that detail: the offset in the partitions level is a valuable information I think

is there a way in kminion to get that same information (the offset reached by a consumer group in the level of partitions of one topic) ?

Thanks in advance

[weeco]

Yes there, is the following metric:

# HELP kminion_kafka_consumer_group_topic_partition_lag The number of messages a consumer group is lagging behind the latest offset of a partition
# TYPE kminion_kafka_consumer_group_topic_partition_lag gauge
kminion_kafka_consumer_group_topic_partition_lag{group_id="bigquery-sink",partition_id="10",topic_name="shop-activity"} 147481

see: https://github.com/redpanda-data/kminion/blob/master/docs/metrics.md . The metric you mentioned is about the number of offset commits.

[mdkouki]

yes indeed it could be computed like this :
kminion_kafka_topic_partition_high_water_mark - kminion_kafka_consumer_group_topic_partition_lag

Thanks.