This is the code used for the paper Towards Learning Universal Hyperparameter Optimizers with Transformers (NeurIPS 2022).
All base dependencies can be installed from requirements.txt. Afterwards, T5X must be manually installed.
Pre-trained OptFormer as a Policy (Example Notebook)
To use our OptFormer pre-trained individually on public BBOB and HPO-B benchmarks, follow the steps:
- (Optional) Download the model checkpoint from
gs://gresearch/optformer/model_checkpointsfor faster loading. - Load the model checkpoint into the
InferenceModel, as shown in policies_test.py.
The InferenceModel will then be wrapped into the OptFormerDesigner, which follows the same API as a OSS Vizier standard Designer.
To train an OptFormer model, the data will need to consist of a large collection of studies. This data may come from two sources:
- Our generated dataset, in the form of a Tensorflow Dataset (TFDS), which can be found in [TODO]. Training using this data converged to the model checkpoint found above.
- Custom user-generated studies, which can be done using OSS Vizier's benchmarking pipeline.
Numerical results used in the paper (for e.g. plotting and comparisons) can be found in the results folder.
If you found this codebase useful, please consider citing our paper. Thanks!
@inproceedings{optformer,
author = {Yutian Chen and
Xingyou Song and
Chansoo Lee and
Zi Wang and
Qiuyi Zhang and
David Dohan and
Kazuya Kawakami and
Greg Kochanski and
Arnaud Doucet and
Marc'Aurelio Ranzato and
Sagi Perel and
Nando de Freitas},
title = {Towards Learning Universal Hyperparameter Optimizers with Transformers},
booktitle = {Neural Information Processing Systems (NeurIPS) 2022},
year = {2022}
}
Disclaimer: This is not an officially supported Google product.