MLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility, deployment, and a central model registry. MLflow currently offers four components:
- MLflow Tracking :
Record and query experiments: code, data, config, and results
[https://www.mlflow.org/docs/latest/tracking.html] - MLflow Projects :
Package data science code in a format to reproduce runs on any platform.
[ https://www.mlflow.org/docs/latest/projects.html ] - MLflow Models :
Deploy machine learning models in diverse serving environments.
[ https://www.mlflow.org/docs/latest/models.html] - Model Registry :
Store, annotate, discover, and manage models in a central repository.
[ https://www.mlflow.org/docs/latest/model-registry.html ]
The MLflow Tracking component is an API and UI for logging parameters, code versions, metrics, and output files when running your machine learning code and for later visualizing the results. MLflow Tracking lets you log and query experiments using Python, REST, R API, and Java API APIs.
- Aws Account with billing setup
- Backend Store [Mysql Server]: It stores (runs, parameters, metrics, tags, notes, metadata, etc)
- Artifacts Store [S3 Bucket ]: It stores (files, models, images, in-memory objects, or model summary, etc).
region:
name: ap-south-1
machine-configuration:
platform: ubuntu-jammy-22.04
storage: /dev/sda1 [50 GB]
instance_type: t2.small/medium/large [as per required]
port-to-open: 8000 TCP port
ip-range: 0.0.0.0/0
key-pair:
name: mlflow
store: somewhere safe
region:
name: ap-south-1
machine-configuration:
mysql-version: 8.0.30
tier: as-per-required
username: admin
password: open-source
storage: as-per-required
auto-scaling: enable
vpc: default
subnet: default
public-access: Yes
port: 3306
additional-configuration:
database_name: mlflowdb
security:
port-to-open: all traffic
ip-range: 0.0.0.0/0
local:
mysql-workbench:
database-endpoint:
username: machine-configuration[username]
pass: machine-configuration[pass]
dbname: machine-configuration[additional-configuration][database_name]
final: mysql://username:pass@endpoint/DB-name
Navigate to scripts/mlflow-server-setup.sh
1. Upgrade the system
2. Install Aws cli
3. Install anaconda
4. Create new env [ mlflow ]
5. launch mlflow as a service
6. reload the system
Navigate to scripts/authenticate-mlflow.sh
1. Install engnix
2. Create Ngnix service
3. set username and password
4. Restart the ngnix service
5. EXPORT the username and password in client env:
6. Enjoy
we have just completed the production setup of the mlflow enviorment. To scale up automatically use kubernetes or aws Fargate. resource link: https://aws.amazon.com/blogs/machine-learning/managing-your-machine-learning-lifecycle-with-mlflow-and-amazon-sagemaker/