Django app for storing time-series metrics in Elasticsearch.
[Work in progress] Below is a sci-fi API. This isn't working yet!
- Python 2.7 or >=3.6
- Django 1.11 or 2.0
- Elasticsearch 6
Add "elasticseach_metrics"
to INSTALLED_APPS
.
INSTALLED_APPS += [
"elasticsearch_metrics",
]
Define the ELASTICSEARCH_DSL
setting.
ELASTICSEARCH_DSL = {
"default": {"hosts": "localhost:9200"}
}
This setting is passed to elasticsearch_dsl.connections.configure
so
it takes the same parameters.
In one of your apps, define a new metric in metrics.py
.
A Metric
is a subclass of elasticsearch_dsl.Document
.
# myapp/metrics.py
from elasticsearch_metrics import Metric
from elasticsearch_dsl import Integer
class PageView(Metric):
user_id = Integer()
Use the sync_metrics
management command to ensure that the index template
for your metric is created in Elasticsearch.
# This will create an index template called myapp_pageview
python manage.py sync_metrics
Now add some data:
from myapp.metrics import PageView
user = User.objects.latest()
view = PageView(user_id=user.id)
# By default we create an index for each day.
# Therefore, this will persist the document
# to an index called, e.g. "myapp_pageview-2020.02.04"
view.save()
Go forth and search!
# perform a search across all page views
PageView.search()
You can configure the index template settings by setting
Metric.Index.settings
.
class PageView(Metric):
user_id = Integer()
class Index:
settings = {
"number_of_shards": 2,
"refresh_interval": "5s",
}
class PageView(Metric):
user_id = Integer()
class Index:
settings = {
"number_of_shards": 2,
"refresh_interval": "5s",
}
class Meta:
template_name = "myapp_pviews"
template = "myapp_pviews-*"
ELASTICSEARCH_DSL
: Required. Connection settings passed toelasticsearch_dsl.connections.configure
.ELASTICSEARCH_METRICS_DATEFORMAT
: Dateformat to use when creating indexes. Default:%Y.%m.%d
(same dateformat Elasticsearch uses for date math)
sync_metrics
: Ensure that all index templates have been created for your metrics.
python manage.py sync_metrics
clean_metrics
: Clean old data using curator.
python manage.py clean_metrics myapp.PageView --older-than 45 --time-unit days
Signals are located in the elasticsearch_metrics.signals
module.
pre_index_template_create(Metric, index_template)
: Sent beforePUT
ting a new index template into Elasticsearch.
_source
and_all
are disabled by default on metric indices in order to save disk space. For most metrics use cases, Users will not need to retrieve the source JSON documents. Be sure to understand the consequences of this: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html#_disabling_source . To enable_source
, you can override it inclass Meta
.
class MyMetric(Metric):
class Meta:
source = MetaField(enabled=True)
MIT Licensed.