clear out of data metrics
wangxiangyu opened this issue · 5 comments
hi
I would like to collect http request count. So I use Counter and http_host+http_path as labels to do this. As time goes on, some http_host+http_path no longer exist, but when I curl the metric api, the old metrics are still there and there are garbages to me.
So how do I clear out of data metric? Maybe by a reset function to clear all metrics.
This is the correct behaviour, time series should not appear and disappear over time as that's difficult to deal with.
@brian-brazil But this is a normal and natural need. I did this through periodical reset
With metrics this is not normal nor natural. A time series should exists within a process from when the process is created until when it is terminated.
Hey @brian-brazil. Can you explain how would you tackle problem of monitoring ephemeral things?
In my case we would be accumulating kubernetes pods and containers data and as a sysadm you would want your exporter to run forever without restart.
So at the end of the day you are exposing a lot of stale metrics, wasting storage space on your prometheus server and introducing noise. Is this expected behaviour or am I missing something?
When writing an exporter, you want a custom collector: https://github.com/prometheus/client_python#custom-collectors