Evaluate&Store Engine for OpenSLO–defined Objectives
SLO Evaluator is intended to be run at predefined intervals (like a cronjob) and on each perform the following steps:
- (TODO) Read configs defininig data sources and data sinks.
- Read all yaml files from the
SLOs
directory and parse all SLOs defined therein (see example-SLOs dir). - For each SLO defined connect to the data source, retrieve required data, evaluate the SLO for the current time window and (TODO) store the results in the appropriate data sinks.
- Clone this repo.
- Make sure you have all the python modules from
requirements.txt
. - Generate some fake data like so:
./test/datagen.py --id-column-header sensor --ids=cpu,system,gpu --column node=klapek --column instance="localhost:9100" -c 200000 -s 60 -o test/node_temps.csv
- Give it some test files and run the code:
ln -s example-SLOs SLOs ./cronrun
You should see some example SLOs being calculated from the fake data you just generated.
- All configs are in yaml format.
- SLO configs are in the OpenSLO format.
- Deviations from the upstream spec should be kept to a minimum. Working with upstream on extending the spec where needed is emphasized.
- The code is in Python.
- Data manipulation is done using Pandas DataFrames.
- In principle any combination of data source and data sink should be supported.
- However the initial design is based around supporting Prometheus and influences of that choice might be visible in places.
- Unit tests all the things.
- Start with functions that perform calculations, those are critical to be kept stable.
- Set up CI in the project once the tests are up.
- Maybe a linter while we're at it (PEP8 at least?).
- Add
ratioMetric
support. - Fully support Prometheus as a data source.
- Waiting on this prometheus-api-client PR to land first.
- Add support for Vault for auth token retrieval for Prom.
- Spin off source–agnostic calculation code into its own module.
- Add some basic cli options to
cronjob
using Click module.- Log level, defining where configs are, maybe something else.
- Add an initial data sink.
- SQL? Prom using pushgateway?
- Morph
objectiveMetric
in current code into whatever the new metric becomes (discussion).