Benchmark ML Experiment Tracking Tools

This repo is a benchmark for ML experiment tracking tools. We build some ML projects from scratch and upgrade them with different experiment tracking tools. The goal is to provide a detailed comparison of different experiment tracking tools, so users can choose the best one for their projects.

Training and Inference are supported.
Experiment management with
- hydra
- tensorboard
- neptune.ai
- wandb
- mlflow
Various Frameworks and Models
- PyTorch Vision for Scene Classification
- TIMM for Image Classification
- HuggingFace for NLP
Model Zoo with pretrained models

🚀 Installation

# Download the code
git clone git@github.com:MLSysOps/ml_exp_tracking_benchmark.git
cd ml_exp_tracking_benchmark

# Create a conda environment
conda create -n ml_track_benchmark python=3.8
conda activate ml_track_benchmark

# Install dependencies
pip install - r requirements.txt

🏃‍♀️ Quick Start

Please download the data from [Place2 Data]

# 1. Download and unzip the data
sh download_data_pytorch.sh

# 2. Train a model
export PYTHONPATH=$PYTHONPATH:$(pwd)
python benchmark/main_tensorboard.py

🔨 Results

🔥 Neptune.ai

😀 Model Zoo (Pretrained Models)

Please refer [Model Zoo]

🎉 Acknowledge

The dataset and basic code comes from [MIT Place365]

Thanks for the great work!