tarantool/metrics

Summary quantiles respond +Inf if rps decreases to zero

DifferentialOrange opened this issue · 3 comments

If there are 0 new requests for some time, summary collector quantiles respond with +Inf values.

image

http_server_request_latency{path="/hello",method="GET",alias="tnt_router",status="200",quantile="0.5"} +Inf
http_server_request_latency{path="/hello",method="GET",alias="tnt_router",status="200",quantile="0.9"} +Inf
http_server_request_latency{path="/hello",method="GET",alias="tnt_router",status="200",quantile="0.99"} +Inf
http_server_request_latency{path="/goodbye",method="POST",alias="tnt_router",status="500",quantile="0.5"} +Inf
http_server_request_latency{path="/goodbye",method="POST",alias="tnt_router",status="500",quantile="0.9"} +Inf
http_server_request_latency{path="/goodbye",method="POST",alias="tnt_router",status="500",quantile="0.99"} +Inf
http_server_request_latency{path="/hell0",method="GET",alias="tnt_router",status="400",quantile="0.5"} +Inf
http_server_request_latency{path="/hell0",method="GET",alias="tnt_router",status="400",quantile="0.9"} +Inf
http_server_request_latency{path="/hell0",method="GET",alias="tnt_router",status="400",quantile="0.99"} +Inf

It breaks visualization with ugly vertical lines. I'm sure there are better approaches to handle this.

Reproducer

You may use https://github.com/tarantool/grafana-dashboard docker cluster and stop load container after several minutes.

docker-compose up
# let it generate some metrics
docker stop grafana-dashboard_load_generator_1

Now it's a part of the quantile algorithm and covers with tests:

g.test_query_on_empty_quantile = function()

We could convert values from empty quantile collector to null.

Now it's a part of the quantile algorithm and covers with tests:

g.test_query_on_empty_quantile = function()

We could convert values from empty quantile collector to null.

I think it is a good idea

function stream:query(q)

if s.n == 0 then
  return nil
end

Since stream:query function is called after buffer flush, if stream is empty (== 0) then there was no insertion in collector.

@DifferentialOrange