Yelrose/optuna-examples

Examples for https://github.com/optuna/optuna

Optuna Examples

This page contains a list of example codes written with Optuna.

Simple Black-box Optimization

• Quadratic function

Examples with ML Libraries

• AllenNLP
• AllenNLP (Jsonnet)
• Catalyst
• CatBoost
• Chainer
• ChainerMN
• Dask-ML
• FastAI V1
• FastAI V2
• Haiku
• Gluon
• Keras
• LightGBM
• LightGBM Tuner
• MXNet
• PyTorch
• PyTorch Ignite
• PyTorch Lightning
• PyTorch Lightning (DDP)
• RAPIDS
• Scikit-learn
• Scikit-learn OptunaSearchCV
• Scikit-image
• SKORCH
• Tensorflow
• Tensorflow (eager)
• XGBoost

An example where an objective function uses additional arguments

The following example demonstrates how to implement an objective function that uses additional arguments other than trial.

• Scikit-learn (callable class version)

Examples of Pruning

The following example demonstrates how to implement pruning logic with Optuna.

• Simple pruning (scikit-learn)
• Pruning for CatBoost

In addition, integration modules are available for the following libraries, providing simpler interfaces to utilize pruning.

• Pruning with Catalyst integration module
• Pruning with Chainer integration module
• Pruning with ChainerMN integration module
• Pruning with FastAI V1 integration module
• Pruning with FastAI V2 integration module
• Pruning with Keras integration module
• Pruning with LightGBM integration module
• Pruning with MXNet integration module
• Pruning with PyTorch integration module
• Pruning with PyTorch Ignite integration module
• Pruning with PyTorch Lightning integration module
• Pruning with PyTorch Lightning integration module (DDP)
• Pruning with Tensorflow integration module
• Pruning with XGBoost integration module
• Pruning with XGBoost integration module (cross validation, XGBoost.cv)

Examples of Samplers

• Warm Starting CMA-ES

Examples of User-Defined Sampler

• SimulatedAnnealingSampler

Examples of Multi-Objective Optimization

• Optimization with BoTorch
• Optimization of MLP with PyTorch

Examples of Visualization

• Visualizing study

An example to enqueue trials with given parameter values

• Enqueuing trials with given parameters

Examples of MLflow

• Tracking optimization process with MLflow

Examples of Weights & Biases

• Tracking optimization process with Weights & Biases

Examples of Hydra

• Optimization with Hydra

Examples of Distributed Optimization

• Optimizing on Kubernetes
• Optimizing with Ray's joblib backend

Examples of Reinforcement Learning

• Optimization of Hyperparameters for Stable-Baslines Agent

External projects using Optuna

• Allegro Trains
• BBO-Rietveld: Automated crystal structure refinement
• Catalyst
• CuPy
• Hydra's Optuna Sweeper plugin
• Mozilla Voice STT
• neptune.ai
• OptGBM: A scikit-learn compatible LightGBM estimator with Optuna
• PyKEEN
• RL Baselines Zoo
• Hyperparameter Optimization for Machine Learning, code repository for online course

PRs to add additional projects welcome!

Running with Optuna's Docker images?

You can use our docker images with the tag ending with -dev to run most of the examples. For example, you can run PyTorch Simple via docker run --rm -v $(pwd):/prj -w /prj optuna/optuna:py3.7-dev python pytorch/pytorch_simple.py. Also, you can try our visualization example in Jupyter Notebook by opening localhost:8888 in your browser after executing this:

docker run -p 8888:8888 --rm optuna/optuna:py3.7-dev jupyter notebook --allow-root --no-browser --port 8888 --ip 0.0.0.0 --NotebookApp.token='' --NotebookApp.password=''