Implement the *Ultimate Optimizer*
Closed this issue · 1 comments
joelberkeley commented
This Nuerips 2022 awards paper introduces the Ultimate Optimizer, which allows us to use autodiff to find hyperparameters, which could dramatically simplify (and speed up?) ML workflows. Implement it.
Requires SGD first?
joelberkeley commented
Paper review suggested that while it was good, it wasn't as useful as it might have sounded. Let's just stick with sgd and adam for now. I'll leave this open as an interesting future goal