Before beginning this tutorial, ensure you have created an account on the trainML platform and you have Docker installed.
This tutorial should cost less than 180 credits ($18) if you use the small or medium instance type and the same setting as the guide. Tuning takes approximately 15 hours using 4 medium GPUs or 30 hours using 4 small GPUs.
This tutorial uses the MXNet Faster-RCNN object detection example to perform a parallelized hyperparameter tuning job on PASCAL VOC using the hyperopt library.
- Clone this repo to a local directory and place it in a new python virtual environment
- Install the prerequisites
pip install -r requirements.txt - Prepare the local directory and download the dataset:
mkdir mongodb
python pascal_voc.py
rm ~/.mxnet/datasets/voc/*.tar
rm ~/.mxnet/datasets/voc/benchmark.tgz
- In a new terminal window, navigate to the root of the repository and run:
docker run -v $(pwd)/mongodb:/data/db -p 27017:27017 mongo
- In the previous terminal window, run
python tune.py - Create a job on the trainML platform with the settings documented in the tutorial guide
- Connect to the job by following the instructions in the getting started guide in a separate terminal window.
You must keep the trainML connect utility, the
python tune.pyscript, and the MongoDB docker container running for the full duration of the experiment. If any one of the 3 stop, the workers will no longer be able to report their results and find new workloads and will eventually terminate.
- Once all 50 trials are evaluated, the final statement should print out the results of the experiment,