Elasticsearch Datasource for loudml-grafana
fadjar340 opened this issue · 10 comments
Dear Sir,
Is it possible to use elasticsearch as datasource for loudml?
Is there some example for this?
Regards,
Fadjar340
Good day @fadjar340 ,
LoudML server supports Elasticsearch (self hosted and AWS) as a datasource
https://github.com/regel/loudml/blob/develop/loudml/elastic.py
Let me add a parser into "Create Baseline" button to handle Elasticsearch on Grafana side.
Will keep you posted.
With best regards,
Volodymyr
You rock @vsergeyev !!!
Fadjar340
You welcome @fadjar340 !
if it possible - my you add a note about index structure/fields you use,
and query in Grafana datasource.
I just use the standard index structure from elasticsearch, the query will follow the index structure.
It's possible to use all the index structure because a lots of metric, log and audit system that can be use and forecast.
There several standard indices, such as filebeat, metricbeat, auditbeat, apm, heartbeat, winlogbeat, elastic use some common structure to build this indices, so I'll try to get the standard index structure in my system, then I'll send here...
Those i mentioned is the common index that have by elasticsearch.
I'll send when it's ready...
Dear @vsergeyev ,
I attached the standard index structure of the elasticsearch.
elasticsearch_index_structure.tar.gz
@fadjar340 cool, thank you for this data!
You're welcome @vsergeyev
@fadjar340 I try to follow this index structure. Looks good so far.
If it is a chance you may share with me also 1-2 queries you use in Grafana panel.
It may speedup my work.
Thank you in advance,
V.
Dear @vsergeyev ,
Below the sample query for elasticsearch from grafana:
request:Object
url:"api/datasources/proxy/10/_msearch"
method:"POST"
data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"logstash-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586624400000,"lte":1586706446964,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"serialnum: (\"18b16902b65c\")"}}]}},"aggs":{"5":{"terms":{"field":"sonicwall.wan.ip","size":500,"order":{"_count":"desc"},"min_doc_count":1},"aggs":{"2":{"date_histogram":{"interval":"30s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586624400000,"max":1586706446964},"format":"epoch_millis"},"aggs":{"1":{"avg":{"field":"sonicwall.source.bytes","script":{"inline":"_value*8"}}}}}}}}} "
request:Object
url:"api/datasources/proxy/21/_msearch?max_concurrent_shard_requests=5"
method:"POST"
data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"max":{"field":"rtt.us"}}}}}} {"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"avg":{"field":"rtt.us"}}}}}} {"search_type":"query_then_fetch","ignore_unavailable":true,"index":"heartbeat-*"} {"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1586685034661,"lte":1586706634661,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"monitor.name: food AND url.domain: 10.124.138.1"}}]}},"aggs":{"2":{"date_histogram":{"interval":"10s","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":1586685034661,"max":1586706634661},"format":"epoch_millis"},"aggs":{"1":{"min":{"field":"rtt.us"}}}}}} "