Apigee metrics from GCP - bucket type prometheus metrics value is not increasing monotonically
sundar-ka opened this issue · 6 comments
We've been introduced to this stackdriver_exporter recently and started using it to export metrics from GCP infra to our prometheus. Specifically the metrics of Apigee proxy provided as a managed service in GCP. We found a specific type of metric which got exported through the stackdriver exporter is not increasing monotonically in its value over the time. This metric type is used to bucket the range of values in different buckets which then can be used to calculate quantiles using histogram_quantile function with prometheus.
Here is an example for metric which is used for calculating latency whose value didn't increase monotonically
![test](https://private-user-images.githubusercontent.com/78709553/331630994-16a385a1-ed53-4861-934b-3cf56958f705.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY2NTg2MDIsIm5iZiI6MTcxNjY1ODMwMiwicGF0aCI6Ii83ODcwOTU1My8zMzE2MzA5OTQtMTZhMzg1YTEtZWQ1My00ODYxLTkzNGItM2NmNTY5NThmNzA1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI1VDE3MzE0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQ5YTliNDhlNDM3OWIzOGM2MWM3ZWE0NzUzZjg1YjVkNzA1OWI0Y2Y1OWYxNTY5MjNhOWVjYTE2NGU5YjZjZmImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.9y82hhG0OTSUy56LkctlSRIy0KV58Tk_s-iTHfZb6iM)
Error got from prometheus while calculating histogram quantile using the above metric
![test1](https://private-user-images.githubusercontent.com/78709553/331632758-7be78d3d-49a2-42ec-9181-b232f19666cf.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY2NTg2MDIsIm5iZiI6MTcxNjY1ODMwMiwicGF0aCI6Ii83ODcwOTU1My8zMzE2MzI3NTgtN2JlNzhkM2QtNDlhMi00MmVjLTkxODEtYjIzMmYxOTY2NmNmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI1VDE3MzE0MlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE4MmM2ZmJhOTY0NTFkY2FiMmQ4MWJjODBkNmVkYThlN2FkYWU2MjMzNDkzNDhhN2QxZGQ3N2FmMzU4ZGQwYWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.p2CL5O0d-5dB23eAXZeF4OOnOeKAdQCvT9osO5OnmVE)
This is a helm based deployment in a kubernetes cluster.
Helm Chart version deployed - v4.5.0
Exporter app version deployed - v0.15.0
Expected:
The metric _bucket value has to be increasing always like a counter type metric
Help needed . Thanks