DataDog/the-monitor

Template Elasticsearch Timeboard indexing/query latency might be wrong

yaoxiaotong opened this issue · 4 comments

The template chart of indexing latency and query latency might be wrong. It shows different with Kibana.

Take query latency for example. The template is using rate of fetch time and rate of query time like this:
image

It might be better to use derivatives to get the same chart with Kibana like this:
(derivative of fetch time + derivative of query time ) / derivative of query total

image

The json of this is

{ "viz": "timeseries", "status": "done", "requests": [ { "q": "( derivative(sum:elasticsearch.search.fetch.time{$cluster}) + derivative(sum:elasticsearch.search.query.time{$cluster}) ) * 1000 / derivative(sum:elasticsearch.search.query.total{$cluster})", "aggregator": "avg", "conditional_formats": [], "type": "line" } ], "autoscale": true }

Thanks for the report @yaoxiaotong. Can you take a look @echang26 ?

Hi @yaoxiaotong thanks for bringing this to our attention. We used your suggestion to apply a fix to our default Elasticsearch timeboard's indexing and query latency graphs, and it does look like those graphs are now more similar to the ones shown in Marvel/Kibana (see below).

screen shot 2017-06-27 at 4 06 18 pm

Thanks for letting us know.

@johnaxel @echang26 thanks for your response and fixing. This solved the problem.

Thanks for the report!