when burrow expires a consumer group, the exporter does not (always?) remove it

[ehthayer]

The blue rectangle in the attached graph is one of these cases. It stops after burrow exporter was bounced.

[elukey]

I can confirm the issue, happened the same to me as well..

Seems to be an issue with the Prometheus libs, a found a similar use case in prometheus/client_python#182. By design the behavior outlined in this issue seems to be the expected one, namely metrics cannot appear/disappear during the lifetime of the exporter process. I have no idea if there is a workaround for the go_client, but I'd add a not in the documentation explaining the problem so others will be aware.

[jirwin]

I think this is part of the nature of prometheus. I don't know of any way to remove an individual timeseries from the exporter(at runtime). We can unregister the collector, but that is not what we want to do.

I'll look into it a bit more and see if there is anything we can do.

[shamil]

If still relevant, I've refactored the exporter to use prometheus custom collector implementation, which will solve the issue with stale metrics.

Since it's a refactor my repo was hard-forked from this one. Thanks @jirwin for your work.

[whaike]

I removed one Topic from kafka ,but burrow-exporter still have old topic metrics with unchanging values untill reboot burrow-exporter.

[saumyadoogar]

Im also facing this issue, its fairly common to see consumers that no longer exist,
restarting the exporter fixes it.