Summary quantiles respond +Inf if rps decreases to zero
DifferentialOrange opened this issue · 3 comments
If there are 0 new requests for some time, summary collector quantiles respond with +Inf values.
http_server_request_latency{path="/hello",method="GET",alias="tnt_router",status="200",quantile="0.5"} +Inf
http_server_request_latency{path="/hello",method="GET",alias="tnt_router",status="200",quantile="0.9"} +Inf
http_server_request_latency{path="/hello",method="GET",alias="tnt_router",status="200",quantile="0.99"} +Inf
http_server_request_latency{path="/goodbye",method="POST",alias="tnt_router",status="500",quantile="0.5"} +Inf
http_server_request_latency{path="/goodbye",method="POST",alias="tnt_router",status="500",quantile="0.9"} +Inf
http_server_request_latency{path="/goodbye",method="POST",alias="tnt_router",status="500",quantile="0.99"} +Inf
http_server_request_latency{path="/hell0",method="GET",alias="tnt_router",status="400",quantile="0.5"} +Inf
http_server_request_latency{path="/hell0",method="GET",alias="tnt_router",status="400",quantile="0.9"} +Inf
http_server_request_latency{path="/hell0",method="GET",alias="tnt_router",status="400",quantile="0.99"} +Inf
It breaks visualization with ugly vertical lines. I'm sure there are better approaches to handle this.
Reproducer
You may use https://github.com/tarantool/grafana-dashboard docker cluster and stop load container after several minutes.
docker-compose up
# let it generate some metrics
docker stop grafana-dashboard_load_generator_1Now it's a part of the quantile algorithm and covers with tests:
metrics/test/quantile_test.lua
Line 144 in 7652824
We could convert values from empty quantile collector to null.
Now it's a part of the quantile algorithm and covers with tests:
metrics/test/quantile_test.lua
Line 144 in 7652824
We could convert values from empty quantile collector to
null.
I think it is a good idea
Line 158 in 7652824
if s.n == 0 then
return nil
endSince stream:query function is called after buffer flush, if stream is empty (== 0) then there was no insertion in collector.
