Mike

Mycroft Holmes (High-Optional, Logical, Multi-Evaluating Supervisor) aka Mike.

The tool that collects metrics for software components based on:

bug reports / tickets from JIRA
features usage statistics taken from Google Analytics or AWS Athena
access and application logs from Elasticsearch
any data that can be queried from MySQL database

See the list of all available sources.

Install

virtualenv -ppython3 env
. env/bin/activate
pip install -e .

To install development dependencies run make inside virtualenv.

Set up a config file

Please refer to test/fixtures/config.yaml and README.md in mycroft_holmes/source directory and prepare your own config file.

Save it and store its path in MIKE_CONFIG env variable. It is used by the front-end app and metrics collector script.

Otherwise you'll get:

AssertionError: Please specify where your config YAML file is in MIKE_CONFIG env variable.

Running UI

Mycroft Holmes comes with Flask-powered web-application that provides a dashboard with an overview of components and their metrics.

Run the following to try it out in development mode:

make server_dev

Now visit /version.json.

Mike uses python-dotenv. Env variables specified in .env in your working development directory will be loaded automatically.

Collecting metrics

Let's assume that this repository has been cloned into /home/macbre/github/Mike and virtual env has been set up. Now add the following to your crontab:

SHELL=/bin/bash
# m h  dom mon dow   command
2 2,14 *   *   *     ( cd /home/macbre/github/Mike && source env/bin/activate && source .env && collect_metrics test/fixtures/config.yaml ) >> /home/macbre/Mike.log 2>&1

Using Docker

You can use our official Docker image:

docker pull macbre/mike:latest
docker run -p5000:5000 -it mike

Run the following command periodically to keep metrics up to date:

docker run -it mike collect_metrics

Passing your custom YAML config file

By default Mike docker container will use a sample config file located in /example.yaml. You should use your own. Please refer to "Set up a config file" section above.

Assuming that you have a local /home/mike/config/.env file with all your specific credentials that are referenced in /home/mike/config/mike.yaml config. Run the following:

docker run -v /home/mike/config:/opt/config -p5000:5000 --env-file /home/mike/config/.env -e MIKE_CONFIG=/opt/config/mike.yaml -it mike

.env file should follow this convention:

DATABASE_USER=mike_db
DATABASE_PASSWORD=d97b4e7998a07bd1b2da4c21f29ec183ad3eec20

Scripts

The following command line scripts are available in Mike's virtual environment:

`collect_metrics`

This script should be run periodically to collect metrics for features defined in YAML config file.

Simply run collect_metrics <path to YAML config file>

Environment variables passed to the script will be used to replace variables in the YAML config file.

For instance, running the following:

DATABASE_USER=foo DATABASE_PASSWORD=2505eb2474b2 collect_metrics test/fixtures/config.yaml

will replace ${...} placeholders with appropriate values taken from environment variables provided above.

sources:
  - name: wikia/tags-report
    kind: common/mysql
    host: db.prod
    user: "${DATABASE_USER}"
    password: "${DATABASE_PASSWORD}"

List of all available sources with full documentation.

`generate_source_docs`

Prints out Markdown with sources documentation taken from the code, to be pasted into mycroft_holmes/sources/README>md when a new source is added or an existing one is updated.

Config file

An example YAML config file can be found in test/fixtures directory.

Mike's architecture, sources structure and configuration are described in mycroft_holmes/sources directory.

License

Dashboard sidebar's background image is used under public domain license. Favicon made by Freepik is licensed by CC 3.0 BY.

Tests

After setting up virtual env, please install all dependencies (including dev ones) via make init. Then run:

make test

MySQL storage integration tests

TEST_DATABASE env variable needs to be set to run these tests. schema.sql and schema_test.sql files need to be applied as well. Please refer to .travis.yml when in doubt.

You may want to include storage integration tests. They do require running MySQL server. Connection details can be provided via .env file:

$ cat .env
TEST_DATABASE=mycroft_holmes
TEST_DATABASE_USER=foo
TEST_DATABASE_PASSWORD=bar

$ make test

# access mysql console with the above credentials
$ make mysql_cli

Please note that tests truncate the tables.

Wikia/Mike