Bayesian Optimization with noisy observations using Expected Improvement
BrunoKM opened this issue · 3 comments
This issue is about what the proper way of dealing with noisy observations in Bayesian Optimization with EmuKit is at the moment? Specifically, how is the user expected to deal with noisy observations when using the Expected Improvement acquisition?
Currently, the ExpectedImprovement acquisition in EmuKit assumes noiseless observations of the function Y:
ExpectedImprovement(x) = Y_min - E[min(y(x), Y_min)]
There are some common heuristics that allow Expected Improvement to be used with noisy observations, likely the most popular one being the one described in Picheny et al. 2012 where the the Expected Improvement is calculated with respect to noiseless predictions from the model, and the previous minimum Y_min is taken as the smallest mean of the posterior GP predictions at the observed points.
To the best of my knowledge, there is currently no simple way to deal with noisy observations with the Expected Improvement acquisition function in EmuKit at present. Is this something that remains to be implemented, or are there some work-arounds that the user is expected to follow?
You are quite right, Emukit's EI doesn't support noisy observations at the moment. GPy definitely supports noisy observations, so model is the only level where this is supported.
"remains to be fixed" is probably too strong phrasing, as this is the first time this feature really surfaced. But we didn't have immediate plans to implement it, if that's what you mean. Would gladly consider a PR though!
You're right, what I meant was closer to "remains to be implemented", I changed the issue accordingly. I'd be happy to make a PR with this change!
I was wondering, what would be the best way to go about this on the model-wrapping API front? A model that would be compatible with this approximation should be able to predict without noise. This call could be made to the wrapper with an additional keyword arg: emukit_model.predict(X, with_noise=False), but since this is not the standard API of IModel, what's the best way to specify that a model's predict function has this signature? I'm thinking a new interface like IModelWithNoise (or a similar name) would work, but I don't know if that's too small of a difference to have a separate interface class for it. It'd be great to hear your thoughts on this.
As for whether to use this heuristic for dealing with noise, I was thinking of adding a flag in the ExpectedImprovement's __init__ function. If you think a separate acquisition would be a better solution, let me know.
Yes, separate interface is the way to go. This change is the exact reason we've created them. By the same token I'd prefer to see a separate EI class for this, otherwise it would be tricky to specify which model is expected to be used with this acquisition