Utility functions: Posterior mean and epsilon-greedy-based methods
georgedeath opened this issue · 8 comments
Thank you for writing this excellent book, I think it'll become the de facto reference for BO in the future. Appendix D, the annotated bibliography of applications will come in particularly handy for future projects.
Description of issue
Directly using the posterior mean as an acquisition function can prove to be quite powerful (e.g. [1, 2]), due to some unintended exploration taking place because of, for example, the necessarily poor surrogate modelling. Additionally, ε-greedy-based methods [2], where the posterior mean is maximised (1-ε) proportion of the time, and an exploratory move (random selection or selection from the exploration/exploration Pareto front [2]) is performed the rest of the time (ε), have been shown to outperform the standard acquisition functions (e.g. EI, PI, UCB) on a wide variety of problems in the sequential [2], batch [3] and asynchronous [4] settings.
Proposed resolution
It would be good to see a mention of where the posterior mean has been successfully used in BO (e.g. [1,2]), while, of course, acknowledging that if our surrogate model was perfect then it would quickly become stuck (as is currently stated in sec 7.10).
Similarly, a short mention of the less rigorous, but highly effective, epsilon-greedy style methods would also provide the readers with another strategy for artificially encouraging exploration. The work of [2] presents BO directly as a trade-off between exploration and explorations. Its discussions and subsequent epsilon-greedy algorithm would fit in nicely in the section on "Artificially encouraging exploration" (page 153).
Disclaimer: I am the author of references 2-4.
[1] Frederik Rehbach, Martin Zaefferer, Boris Naujoks, Thomas Bartz-Beielstein. 2020. Expected Improvement versus Predicted Value in Surrogate-Based Optimization. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 868–876. Association for Computing Machinery, 2020.
[2] George De Ath, Richard M. Everson, Alma A. M. Rahat, and Jonathan E. Fieldsend. Greed is good: Exploration and exploitation trade-offs in Bayesian optimisation. ACM Transactions on Evolutionary Learning and Optimization, 1(1), 2021.
[3] George De Ath, Richard M. Everson, Jonathan E. Fieldsend, and Alma A. M. Rahat. ε-shotgun: ε-greedy batch Bayesian optimisation. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 787–795. Association for Computing Machinery, 2020.
[4] George De Ath, Richard M. Everson, and Jonathan E. Fieldsend (2021). Asynchronous ϵ-Greedy Bayesian Optimisation. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence.
Thank you for this; I think augmenting §7.10 with a pointer in this direction is a great idea.
Thank you for this; I think augmenting §7.10 with a pointer in this direction is a great idea.
That's great, thank you, Roman. If there's anything I can do to help, please let me know.
I added a reference to your [2] and a tiny bit of discussion in the most recent draft. Please take a look! (Also it's pretty awesome that you managed to publish the very first paper in the first issue of the first volume of a journal!)
Thanks, Roman. I'll take a look when the repository updates -- it doesn't seem to be up yet. (We quite enjoyed that fact too!)
Oops sorry it should be up now.
Roman, the paragraph looks good to me -- thank you including it.
@georgedeath a related question -- do you alphabetize your surname under "D" or "A?"
@georgedeath a related question -- do you alphabetize your surname under "D" or "A?"
I count "De Ath" as the entire surname, so it'd come under "D". Not sure why my family capitalises the "D" though! Thank you for asking @bayesoptbook 👍