xptl

Experiment Tool for Hyper Parameter Searches

Requirements

Python 3.6

Installation

You can install the package directly from GitHub using pip:

pip install git+https://github.com/titus-leistner/xptl.git

Usage

To define your command line arguments, create one or several ini-files. Here is an example:

# inis/config1.ini

[meta]
execute=no

[training]
learning_rate=0.01
batch_size=128

The batch_size is defined in our "parent" ini-file, therefore it is the same for all experiments in this example. However, we want to try two different loss functions. We therefore create two additional files:

# inis/config2.ini

[meta]
parent=config1.ini
prefix=L1 Experiment

[training]
loss=L1

and

# inis/config3.ini

[meta]
parent=config1.ini
prefix=MSE Experiment

[training]
loss=MSE
learning_rate=0.0001

Note, that we include parent=config1.ini to inherit all arguments from our first file. Both children inherit the batch_size, but define a different loss parameter. For the MSE loss we notice that our learning rate is way too high, we therefore override the inherited learning_rate and try a lower number.

Now we can either execute one experiment by calling e.g. xptl inis/config3 or all experiments at once with xptl inis/. As our "parent" ini-file should not execute an experiment, we add execute=no.

To run your experiments, use the following syntax:

xptl-schedule INPUT [BATCH_CMD] [JOB_FILE]

INPUT is the path to your ini-file or a directory containing ini-files. BATCH_CMD is the command that queues a job for your scheduler. Default is sbatch for SLURM. JOB_FILE is your job script. Default is job.sh.

Using the script on our exemplary directory of ini-files runs the following commands:

sbatch -J L1_Experiment-training_learning_rate=0.01-training_batch_size=128-training_loss=L1 job.sh L1_Experiment-training_learning_rate=0.01-training_batch_size=128-training_loss=L1  --training_learning_rate=0.01 --training_batch_size=128 --training_loss=L1

sbatch -J MSE_Experiment-training_learning_rate=0.0001-training_batch_size=128-training_loss=MSE job.sh MSE_Experiment-training_learning_rate=0.0001-training_batch_size=128-training_loss=MSE  --training_learning_rate=0.0001 --training_batch_size=128 --training_loss=MSE

The script generates a job name, either defined as name in the [meta] section, or containing all parameters without spaces. Optionally a prefix to this jobname can be defined as prefix in the [meta] section. The -J argument passes this name to your scheduler. It is also passed to your job script as a first argument followed by all other hyperparameters as additional arguments.

An exemplary job file could look like this:

#!/bin/bash

# Some parametes for the scheduler:
# SBATCH --time=120:00:00
# ...
 
# Create output directory for your experiment
mkdir output/$1

# Start your experiment passing all your parameters to a python script using $@
python experiment.py --output_dir=output/$@

echo exiting

exit 0

Note that in this example, the job name gets passed to the experiment as a path for output files in order to keep an organized structure.