Implementation of some of the very effective tools for training Deep Learning (DL) models that I came across while doing the fastai course on Practical Deep Learning for Coders.
The tools were first presented in the following papers by Leslie N. Smith:
- LR Finder: A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- One Cycle Scheduler: Cyclical Learning Rates for Training Neural Networks
My implementations are a port of the code in fastai library (originally, based on Pytorch) to Keras and are heavily inspired by some of earlier efforts in this direction:
Here's another article I referred to: How Do You Find A Good Learning Rate by Sylvain Gugger of fastai which provides an intuitive understanding of how fastai's LR finder works.