Seizure Detection pipeline

This project aims to use machine learning algorithms to detect seizure from ECG data, in a MLOps environment. Here is an overview of the automated ML pipeline :

This pipeline runs inside a dockerised environment, represented below :

Prerequisites

Dependencies

The pipeline requires these packages to run :

Python >= 3.7
pandas == 1.1.5
numpy == 1.19.5
pyEDFlib == 0.1.22
click == 8.0.1
py-ecg-detectors == 1.0.2
wfdb == 3.4.0
biosppy == 0.7.3
hrv-analysis == 1.0.4
ecg-qc == 1.0b5
great-expectations == 0.13.25
airflow-provider-great-expectations == 0.0.7
psycopg2-binary == 2.8.6

You can install them in a virtual environment on your machine via the command :

    $ pip install -r requirements.txt

Environment

You need to have docker and docker-compose installed on your machine to set the environment up.

Getting started

Setting up environment and launch docker-compose

After cloning this repository, replace the value of the environment variable DATA_PATH in the env.sh file with the absolute path of the data you are working with.

You can now run these commands :

    $ source setup_env.sh
    $ docker-compose build
    $ docker-compose up airflow-init
    $ docker-compose up -d

Warning: Here are the default ports used by the different services. If one of them is already in use on your machine, change the value of the corresponding environment variables in the env.sh file before running the commands above.

Service	Default port
Postgresql	5432
InfluxDB	8086
Airflow	8080
Grafana	3000
MLFlow	5000
Great expectations (via NGINX)	8082
Flower	5555
Redis	6379

UI

Once the services are up, you can interact with their UI :

Airflow : http://localhost:8080
Grafana : http://localhost:3000
MLFlow : http://localhost:5000
Great expectations : http://localhost:8082
Flower : http://localhost:5555

When required, usernames and passwords are admin.

Executing script separately

First export the python path to access the scripts :

    $ export PYTHONPATH=$(pwd)

You can now execute each Python script separately by running :

    $ python3 <path-to-Python-script> [OPTIONS]

The required options are shown by running the --help option.

Setting down the environment

You can stop all services by running :

    $ docker-compose down

If you add the -v option, all services' persistent data will be erased.

License

This project is licensed under the GNU GENERAL PUBLIC License.

fernandokm/seizure_detection_pipeline