/syne-tune

Large scale and asynchronous Hyperparameter and Architecture Optimization at your fingertips.

Primary LanguagePythonApache License 2.0Apache-2.0

Syne Tune: Large-Scale and Reproducible Hyperparameter Optimization

Release Python Version License Downloads Documentation

Alt Text

Documentation

This package provides state-of-the-art algorithms for hyperparameter optimization (HPO) with the following key features:

  • Wide coverage (>20) of different HPO methods, including:
    • Asynchronous versions to maximize utilization and distributed versions (i.e., with multiple workers);
    • Multi-fidelity methods supporting model-based decisions (BOHB and MOBSTER);
    • Hyperparameter transfer learning to speed up (repeated) tuning jobs;
    • Multi-objective optimizers that can tune multiple objectives simultaneously (such as accuracy and latency).
  • HPO can be run in different environments (locally, AWS, simulation) by changing just one line of code.
  • Out-of-the-box tabulated benchmarks that allows you simulate results in seconds while preserving the real dynamics of asynchronous or synchronous HPO with any number of workers.

Installing

To install Syne Tune from pip, you can simply do:

pip install 'syne-tune[extra]'

or to get the latest version from git:

git clone https://github.com/awslabs/syne-tune.git
cd syne-tune
python3 -m venv st_venv
. st_venv/bin/activate
pip install --upgrade pip
pip install -e '.[extra]'

This installs everything in a virtual environment st_venv. Remember to activate this environment before working with Syne Tune. We also recommend building the virtual environment from scratch now and then, in particular when you pull a new release, as dependencies may have changed.

See our change log to see what changed in the latest version.

Getting started

To enable tuning, you have to report metrics from a training script so that they can be communicated later to Syne Tune, this can be accomplished by just calling report(epoch=epoch, loss=loss) as shown in the example below:

# train_height.py
import logging
import time

from syne_tune import Reporter
from argparse import ArgumentParser

if __name__ == '__main__':
    root = logging.getLogger()
    root.setLevel(logging.INFO)

    parser = ArgumentParser()
    parser.add_argument('--steps', type=int)
    parser.add_argument('--width', type=float)
    parser.add_argument('--height', type=float)

    args, _ = parser.parse_known_args()
    report = Reporter()

    for step in range(args.steps):
        dummy_score = (0.1 + args.width * step / 100) ** (-1) + args.height * 0.1
        # Feed the score back to Syne Tune.
        report(step=step, mean_loss=dummy_score, epoch=step + 1)
        time.sleep(0.1)

Once you have a script reporting metric, you can launch a tuning as-follow:

from syne_tune import Tuner, StoppingCriterion
from syne_tune.backend import LocalBackend
from syne_tune.config_space import randint
from syne_tune.optimizer.baselines import ASHA

# hyperparameter search space to consider
config_space = {
    'steps': 100,
    'width': randint(1, 20),
    'height': randint(1, 20),
}

tuner = Tuner(
    trial_backend=LocalBackend(entry_point='train_height.py'),
    scheduler=ASHA(
        config_space, metric='mean_loss', resource_attr='epoch', max_t=100,
        search_options={'debug_log': False},
    ),
    stop_criterion=StoppingCriterion(max_wallclock_time=15),
    n_workers=4,  # how many trials are evaluated in parallel
)
tuner.run()

The above example runs ASHA with 4 asynchronous workers on a local machine.

Supported HPO methods

The following hyperparameter optimization (HPO) methods are available in Syne Tune:

Method Reference Searcher Asynchronous? Multi-fidelity? Transfer?
Grid Search deterministic yes no no
Random Search Bergstra, et al. (2011) random yes no no
Bayesian Optimization Snoek, et al. (2012) model-based yes no no
BORE Tiao, et al. (2021) model-based yes no no
MedianStoppingRule Golovin, et al. (2017) any yes yes no
SyncHyperband Li, et al. (2018) random no yes no
SyncBOHB Falkner, et al. (2018) model-based no yes no
SyncMOBSTER Klein, et al. (2020) model-based no yes no
ASHA Li, et al. (2019) random yes yes no
BOHB Falkner, et al. (2018) model-based yes yes no
MOBSTER Klein, et al. (2020) model-based yes yes no
DEHB Awad, et al. (2021) evolutionary no yes no
HyperTune Li, et al. (2022) model-based yes yes no
ASHABORE Tiao, et al. (2021) model-based yes yes no
PASHA Bohdal, et al. (2022) random or model-based yes yes no
REA Real, et al. (2019) evolutionary yes no no
KDE Falkner, et al. (2018) model-based yes no no
PBT Jaderberg, et al. (2017) evolutionary no yes no
ZeroShotTransfer Wistuba, et al. (2015) deterministic yes no yes
ASHA-CTS Salinas, et al. (2021) random yes yes yes
RUSH Zappella, et al. (2021) random yes yes yes

The searchers fall into four broad categories, deterministic, random, evolutionary and model-based. The random searchers sample candidate hyperparameter configurations uniformly at random, while the model-based searchers sample them non-uniformly at random, according to a model (e.g., Gaussian process, density ration estimator, etc.) and an acquisition function. The evolutionary searchers make use of an evolutionary algorithm.

Syne Tune also supports BoTorch searchers.

Supported multi-objective optimization methods

Method Reference Searcher Asynchronous? Multi-fidelity? Transfer?
Constrained Bayesian Optimization Gardner, et al. (2014) model-based yes no no
MOASHA Schmucker, et al. (2021) random yes yes no

HPO methods listed can be used in a multi-objective setting by scalarization or non-dominated sorting. See multiobjective_priority.py for details.

Examples

You will find the following examples in examples/ folder illustrating different functionalities provided by Syne Tune:

FAQ and Tutorials

You can check our FAQ, to learn more about Syne Tune functionalities.

Do you want to know more? Here are a number of tutorials.

Blog Posts

Security

See CONTRIBUTING for more information.

Citing Syne Tune

If you use Syne Tune in a scientific publication, please cite the following paper:

"Syne Tune: A Library for Large Scale Hyperparameter Tuning and Reproducible Research" First Conference on Automated Machine Learning, 2022.

@inproceedings{
  salinas2022syne,
  title={Syne Tune: A Library for Large Scale Hyperparameter Tuning and Reproducible Research},
  author={David Salinas and Matthias Seeger and Aaron Klein and Valerio Perrone and Martin Wistuba and Cedric Archambeau},
  booktitle={First Conference on Automated Machine Learning (Main Track)},
  year={2022},
  url={https://openreview.net/forum?id=BVeGJ-THIg9}
}

License

This project is licensed under the Apache-2.0 License.