Compatibility with Prometheus and pull-based approach in general
vilunov opened this issue · 2 comments
Hi there!
I'd like to start exposing tokio runtime metrics as part of my application's prometheus metrics. Unfortunately, there is a number of conceptual differences, which make tokio-metrics not really suitable for this.
Prometheus usually scrapes applications' metrics by calling an HTTP endpoint in equal time intervals. In my practice I've encountered scrape intervals between 15 secs and 5 minutes, it is determined by a trade-off in resolution requirements and available storage resources. In any case, all metric changes between two scrapes are not observable via Prometheus, usually the best practice for that is to implement most metrics as non-decreasing counters and derive frequency properties from that.
Also, since each metric scrape is a network interaction, it can be failed and retried without guarantees that the request really made it through to the process or not. Due to that it's important for a metrics endpoint to be stateless, which is violated in how intervals iterator is implemented. Ideally, there would be no state change at all when retrieving the current state of metrics.
Do you think that tokio-metrics is a good place to implement that kind of stuff or do you believe it targets a different type of metrics here?
Does tokio's own RuntimeMetrics API address your use-case? This crate's iterator-based API is built atop that one.
Let me know if the above guidance helped! In the meantime, I'm going to close this issue.