ARM-software/mango

early stopping ?

asah opened this issue · 9 comments

asah commented

Have you considered support for stopping Mango early if the objective function doesn't improve enough?

Would you be open to a contribution PR?

thanks!

sandeep-iitr commented

Thanks. We will be happy to have suggestions and possible contributions. We are a little busy for the next ~4 days due to a deadline. The early stopping strategies are definitely one of our future planned works, and we plan to integrate it into Mango.

flamby commented

+1

I've done something in the same vein that might be useful like early stopping: displaying current best params every time the progress bar changes. I'm launching the search with a huge num_iteration and stop it when I see a satisfying result

tuner.py
For Bayesian optimizer

237             pbar.set_description("Best score: %s" % np.max(Y_sample))
238             print(hyper_parameters_tried[np.argmax(Y_sample)])

For random

274             pbar.set_description("Best score: %s" % np.max(np.array(Y_sample_list)))
275             print(X_sample_list[np.argmax(Y_sample_list)])
  • 👍1
asah commented

PR submitted - request changes and I'll gladly make them!
#21

  • 👍2
sandeep-iitr commented

Thanks for the PR. Please give us some time to review.

asah commented
tihom commented

@asah Thanks for the PR. Inspired by your suggestion, we are currently developing a feature that would allow early_stopping using custom callback provided by the user. This would allow uses to specify custom criteria for early_stopping like a threshold for the objective functions, elapsed iterations since last improvement etc. We would submit a PR this week and welcome your feedback on it.

asah commented
tihom commented

@asah see the PR #24 and see the test that demos your use case for min improvement secs. Let us know if that resolves your request. We would add some examples in docs once the PR is finalized.

asah commented