Minimum improvement early stopping callback
Opened this issue · 0 comments
Current state
Main training function fold_train_ensemble
uses early stopping by default: if a number of subepochs are passed (patience) without an improvement in the validation loss, then training stops.
Problem
Sometime the validation loss will move to a plateau with a shallow slope; the validation loss continues to decrease, but at rate which minimal impact on performance.
Suggestion
A new callback is introduced to to stop training if the validation loss doesn't decrease by a certain amount (or fraction) after a certain number of epochs. This should ideally automatically scale to the typical loss values for each particular training and should accurately detect plateaus without fine tuning by the user. It could, for instance, monitor the rate of change of loss. MetricLogger
already does something similar by monitoring the loss velocity. It should also be resistant to fluctuations in the loss, which may occur, particularity in heavily weighted data.