Managing Objective Function Evaluation Failures in Ax for MOO
Fa20 opened this issue · 6 comments
I'm using Ax to find the optimal solutions for a MOO problem. However, some suggested parameter values cause simulation failures, making it impossible to calculate the objective function. How can we avoid this in Ax? Specifically, how can we abandon these failed trials and obtain new suggestions? Can you provide an example of how to implement this?
Hi @Fa20 -- assuming you are using AxClient as we typically recommend take not of the trial that failed and simply call ax_client.log_trial_failed(trial_index) with the index of the trial that produced a simulation failure. After doing this you are free to generate new points using ax_client.get_next_trial() and continue the optimization.
@mpolson64 thanks for your answer.
this will be not just one trial and may be many of this trial the simulation will be fallied with some of these suggested values of prams and I thought that using abandon will help not to repeat such these values later ?
@mpolson64 is there any difference between ax_client.log_trial_failure(trial_index=trial_index) and ax_client.log_trial_failed(trial_index)
@mpolson64 could you please keep the issue open since I do not think that is completed from my Side. Thank you so much
I did check and did not find ax_client.log_trial_failed(trial_index) could you please help?
@Fa20 on this happening in multiple trials, you will need to repeatedly call ax_client.log_trial_failure(trial_index=n) for each trial that produces a simulation failure.
On FAILED versus ABANDONED, see #372 (comment) for a detailed explanation. In practice we see most of our users desire the FAILED behavior (i.e. treating errors as potentially transient and not blocking parameterizations from being re-suggested) so we provide the log_trial_failure method on AxClient to enable easy management. If you truly want the ABANDONED behavior you may do the following:
trial = ax_client.get_trial(trial_index=n)
trial.mark_abandoned()
To your question about the difference between ax_client.log_trial_failure(trial_index=trial_index) and ax_client.log_trial_failed(trial_index): This was a typo on my part -- log_trial_failure is the correct method to use.
Closing due to inactivity and a solution being proposed, please feel free to reopen :)