MLCommons™ Algorithmic Efficiency

Installation • Rules • Contributing • License

MLCommons Algorithmic Efficiency is a benchmark and competition measuring neural network training speedups due to algorithmic improvements in both training algorithms and models. This repository holds the competition rules and the benchmark code to run it.

Installation

Create new environment, e.g. via conda or virtualenv:

Python minimum requirement >= 3.7

 sudo apt-get install python3-venv
 python3 -m venv env
 source env/bin/activate

Clone this repository:

git clone https://github.com/mlcommons/algorithmic-efficiency.git
cd algorithmic-efficiency

Install the algorithmic_efficiency package:
```
pip3 install -e .
```
You can also install the requirements for individual workloads, e.g. via
```
pip3 install -e '.[librispeech]'
```
or all workloads at once via
```
pip3 install -e '.[full]'
```
Depending on the framework you want to use (e.g. JAX or PyTorch) you need to install them as well. You could either do this manually or by adding the corresponding options:

JAX (GPU)
```
pip3 install -e '.[jax_gpu]' -f 'https://storage.googleapis.com/jax-releases/jax_releases.html'
```
JAX (CPU)
```
pip3 install -e '.[jax_cpu]'
```
PyTorch
```
pip3 install -e '.[pytorch]' -f 'https://download.pytorch.org/whl/torch_stable.html'
```
Development

To use the development tools such as pytest or pylint use the dev option:
```
pip3 install -e '.[dev]'
```
To get an installation with the requirements for all workloads and development, use the argument [full_dev].

Docker

Docker is the easiest way to enable PyTorch/JAX GPU support on Linux since only the NVIDIA® GPU driver is required on the host machine (the NVIDIA® CUDA® Toolkit does not need to be installed).

Docker requirements

Install Docker on your local host machine.
For GPU support on Linux, install NVIDIA Docker support.
- Take note of your Docker version with docker -v. Versions earlier than 19.03 require nvidia-docker2 and the --runtime=nvidia flag. On versions including and after 19.03, you will use the nvidia-container-toolkit package and the --gpus all flag. Both options are documented on the page linked above.

Setup

Clone this repository:

git clone https://github.com/mlcommons/algorithmic-efficiency.git

Build Docker

cd algorithmic-efficiency/ && sudo docker build -t algorithmic-efficiency .

Run Docker

sudo docker run --gpus all -it --rm -v $PWD:/home/ubuntu/algorithmic-efficiency --ipc=host algorithmic-efficiency

Currently docker method installs both PyTorch and JAX

Running a workload

JAX

python3 submission_runner.py \
    --framework=jax \
    --workload=mnist |\
    --submission_path=baselines/mnist/mnist_jax/submission.py \
    --tuning_search_space=baselines/mnist/tuning_search_space.json

PyTorch

python3 submission_runner.py \
    --framework=pytorch \
    --workload=mnist \
    --submission_path=baselines/mnist/mnist_pytorch/submission.py \
    --tuning_search_space=baselines/mnist/tuning_search_space.json

Rules

The rules for the MLCommons Algorithmic Efficency benchmark can be found in the seperate rules document. Suggestions, clarifications and questions can be raised via pull requests.

Contributing

If you are interested in contributing to the work of the working group, feel free to join the weekly meetings, open issues, and see the MLCommons contributing guidelines.

alpv95/algorithmic-efficiency