facebook/Ax

The same point is evaluated multiple times during Integer Optimization with BO.

lapry34 opened this issue · 5 comments

lapry34 commented

I'm attempting hyperparameter optimization of a neural network, where all variables are integers. The issue arises when the feasible set is very small, causing the same point to be returned and evaluated multiple times.

You can find the code in my repository: bachelor_thesis. In this repository, I have expanded the feasible set to prevent this problem. I am also attaching the data where this issue occurred.
cifar10.json
stats.csv

mgarrard commented

Hi @lapry34, this is expected when the parameter set is quite small, or the optimization configuration is challenging to reach within the given search space, thus causing the bounds to be evaluated multiple times. Expanding the search space, which you did by expanding the feasible set, would have been my recommendation. Is there something else/more specific you'd like help with? Or a reason why this expansion doesn't work in practice?

Having some more context on your goals/application could help us brainstorm with you

lapry34 commented

Hello @mgarrard,

Thank you for your recommendations! =)

I was considering implementing an internal library warning that would trigger when an evaluation point has been already computed. In such cases, the process could move to the nearest unevaluated point. If no such point exists, it might be best to skip the evaluation altogether.

This idea arose during experimentation when I noticed that a significant amount of resources were being consumed by repeated evaluations, which increased the overall execution time.

I managed to work around this issue by checking if a value had already been computed before starting the evaluation. If it had, I simply returned the known value.

However, I believe that incorporating a fix directly into the library would be more beneficial and useful in the future.

Balandat commented

Are your evaluations noisy? If so, then re-evaluating the same point in case of a discrete search space may not be a bug but a feature, as it allows to estimate the noise level.

Also, why are you selecting a discrete set of values for the learning rate in https://github.com/lapry34/bachelor_thesis/blob/main/ax_cifar10_restart.py#L126 ? It would be better to have this be a continuous parameter with log_scale=True.

  • 👍2
lapry34 commented

My evaluations should not be noisy. You are right that re-evaluating should estimate the noise level accurately. Is there an option to inform the API that a function is noisy? Maybe it can help in these situations by prompting a re-evaluation only if noise is detected.

Thank you for your help. My supervisor and I initially chose this learning rate setup, but I will now try implementing your suggestions. =)

mgarrard commented

Hey @lapry3, regarding flags for noiseyness there are a few options:

  1. If you want to specify a metric reading as noiseless you can return a tuple of (mean, sem), substituting 0.0 in for the sem
  2. if you either a) return a float instead of a tuple for mean or b) return numpy.NaN for sem, Ax will infer the noise level

Please feel free to re-open this question, or start a new on if you want further help with your setup :)