As the learning rate is one of the most important hyper-parameters to tune for training convolutional neural networks. In this paper, a powerful technique to select a range of learning rates for a neural network that named cyclical learning rate (https://arxiv.org/abs/1506.01186) was implemented with two different skewness degrees. It is an approach to adjust where the value is cycled between a lower bound and upper bound. CLR policies are computationally simpler and can avoid the computational expense of fine tuning with fixed learning rate. It is clearly shown that changing the learning rate during the training phase provides by far better results than fixed values with similar or even smaller number of epochs.
Data Used in this project can be download from here:https://goo.gl/g4D8H5
The experimental results presented in this work demonstrate the benefits of using cyclical learning rate (CLR) method for unbalanced data sets. These cyclical variations for the learning rate between the base lr and max lr can provide optimal classification results, also in case of highly-skewed unbalanced data sets. Simple LR range test can provide the estimation of the learning rate boundaries avoiding the computational complexity and expense of tuning the classifier model with fixed learning rate. We tried different modes of triangular learning rate with a simple CNN with a few epochs to train and test on unbalanced data set. In comparison with fixed learning rate, all the cases of CLR provided better metrics (AUC, and F1 score). In future Clr could be applied on deeper CNN with different architectures to check its performance.