/MLOps

Primary LanguagePythonMIT LicenseMIT

Table of Contents

Overview

Some videos were produced along the way:

Project structure

/misc/ folder contains experimental artifacts.
/src/ folder contains Flask Server code-base and can be treated as main project output. The code was originally generated byswagger-codegen tool, as described in OpenAPI-Spec.
/src/aiml/ folder contains researcher specific coontent and must be synchronised with actual content.
/src/docker/v4.0.2/ folder contains Dockerfile required to build output image and assotiated with other MLOps scenarios files.

Requirements

Python 3.9+
This example uses the Connexion library on top of Flask.

Setup

In that folder, use the following command (as appropriate to your computer) to create and activate a virtual environment named .venv based on your current interpreter:

# Linux
sudo apt-get install python3-venv    # If needed
python3 -m venv .venv
source .venv/bin/activate

# macOS
python3 -m venv .venv
source .venv/bin/activate
#reconstruct dependencies from pyproject.toml
poetry install 

# Windows
py -3 -m venv .venv
.venv\scripts\activate
#reconstruct dependencies from pyproject.toml
poetry install

Poetry dependencies

For numba, a possible solution would be to set the python property to ">=3.9,<3.11" numba (0.55.1) depends on numpy (>=1.18,<1.22), numba (>=0.55.1,<0.56.0) requires numpy (>=1.18,<1.22).

pip3 install poetry #poetry self update
poetry init
poetry add click connexionpip install connexion[swagger-ui] python_dateutil setuptools flask flask_caching jsonify gunicorn scikit-learn numpy pandas torch torchvision xgboost

#backward compartibility downgrade if required
#poetry add xgboost@1.3.3 numba@0.55.1 torch@1.10.0 colorama --platform linux
#poetry remove faiss-cpu

#dev and test environment
poetry add --dev pylint great_expectations pytest flask_testing coverage nose pluggy py randomize

poetry show --tree
poetry show --latest
poetry update
poetry export -f requirements.txt --output requirements.txt --without-hashes
poetry export -f requirements.txt --output test-requirements.txt --without-hashes --dev

#install requiriments directly (not required in DEV!)
pip3 install -r requirements.txt
pip3 install -r test-requirements.txt

To run the server locally, execute the following from the root directory:

python -m swagger_server

Running with Docker

To run the server on a Docker container, please execute the following from the root directory: To build and run a Docker container, please execute the follow commands below:

  1. In "src", execute:
#run from zhibin_liao folder
docker build -t server_ubuntu:v4.0.1 -f docker/v4/Dockerfile . 
#run from flask-server folder
docker build -t server_ubuntu:v4.0.2 -f docker/v4.0.2/Dockerfile . 

#monitoring
docker image ls -a
docker container ls -a
docker run -d -p 5000:5000 server_ubuntu:v4.0.2
docker run --publish 3710:3710 server_ubuntu:v4.0.1

#stop all containers
docker stop $(docker ps -a -q)
#remove all images
docker rmi -f $(docker images -a -q)

#stop one image
docker stop server_ubuntu:v4.0.2
docker kill server_ubuntu:v4.0.2
docker rm server_ubuntu:v4.0.2
docker system prune --volumes --all

# using compose
docker compose -f docker/v4.0.2/docker-compose.yml up
# if you want to update the config sometimes
docker compose -f docker/v4.0.2/docker-compose.yml up --build --force-recreate
  1. In "src/docker/v4.0.2", execute:
# using compose
docker compose up
docker compose up --build --force-recreate

# use docker build
docker build --tag <image_name> -f Dockerfile.<suffix> ../..
  1. starting up a container
docker run -p 3710:3710 server_ubuntu:v4.0.1
docker run -p 5000:5000 server_ubuntu:v4.0.2

Docker compose and volumes

Persist data in the same host with container (Refer to "docker-compose.yml")

  1. Use docker compose create to explicitly create a volume.
  2. Then docker compose up. In case the configure is not updated, use docker compose up --build .

Testing and quality assurance

  1. activate environment
  2. exclude out of scope artefacts (.coveragerc)
  3. run tests and reports

Unit testing, coverage, and static analysis

#all unit tests
python -u ./swagger_server/test/tester.py
#individual unit test notation: -u FileName -k ClassName.MethodName
python -u ./swagger_server/test/tester.py -k TestPosts.test_connection

# coverage testing
coverage run -m unittest discover 
# generate reports
coverage report
coverage html
# run linter and check output rate
pylint --recursive=y ./aiml --output-format=text >pylint_report_aiml.txt
pylint --recursive=y ./swagger_server --ignore=test,models,model_controller.py >pylint_report_swagger.txt
pylint --recursive=y ./swagger_server/test >pylint_report_test.txt

Integration testing

# find all test_* files and run
pytest --disable-warnings
# run all tests in file
pytest --disable-warnings ./swagger_server/test/test_inference_controller.py
pytest --disable-warnings ./swagger_server/test/test_expectations.py
# run particular test
pytest --disable-warnings ./swagger_server/test/test_expectations.py::test_data
pytest --disable-warnings ./swagger_server/test/test_expectations.py::test_cli

# example of MLOps testing
poetry run dvc pull #download models
poetry run dvc repro #reconstruct staging data, integration test for CHANGED modules
poetry run pytest --disable-warnings ./swagger_server/test/test_inference_controller.py
poetry run pytest --disable-warnings ./swagger_server/test/test_expectations.py

#To launch the integration tests, use tox:
sudo pip install tox
tox
#tox not working for py36 and py37 : InterpreterNotFound -> https://github.com/tox-dev/tox/issues/1477
tox -rvve py39
tox -rvve py39 > ./test_tox.log
python -m test > ./test_python.log

Documentation

Project specification can be found in SPEC.md file explaining domain area of knowledge. Project_Roadmap

What is new

V4.0.1 (8 Feb 2022)

In this update, the modifications are:

  1. adding the DL and XGB models for the cabg and intv (intervention) prediction. The outcome includes an additional cabg or intervention prediction (cabg|intv). Note that the cabg and intv prediction models do not use the following features: phys_ckmb, phys_po2, angiogram.
  2. backward compatible to the V3 predictors.
  3. service paths are renamed. For outcome prediction, the service end points are change from /predict/{dl/xgb} to /predict/outcome/{dl/xgb}, the event end point is changed from /predict/event to predict/event/xgb. The cabg and intervention end points are at /predict/revasc/{dl/xgb}.
  4. only one DL and one XGBoost model is install for cabg and intervention. This means supplying the idx=0 or not (idx=-1) will give the same result.
  5. V4.0.1 inherits the clipping operation in V3.3.2 automatically.

V4.0.2 (27 May 2022)

In this update, the modifications are:

  1. Rebuilt API using OpenAPI(Swagger 2.0) and Flask/Connexion server, implement configuration files, CLI options, and logging.
  2. Migrated to Python 3.9, moved dependencies management from pip to poetry. Update all libraries to the latest versions without conflicts. Separated dev and prod autogenerated requirements.
  3. Backward compatible V3, V4 predictors and unit tests. Support all known service interfaces. Generalized inference protocol. Extended collection unit tests and test coverage reports.
  4. Rebuilt Docker image based on Ubunty20.4 LTS. Reduced size of image file from 6GB to 2GB.
  5. Updated specification: SRS, testing plan, OpenAPI interfaces autogenerated documentation.
  6. Created new scenarios for MLOps based on Docker compose, networks, and volumes. Adopting new use cases based on MLFlow, DVC, DagsHub.