GilesStrong/lumin

Minimum improvement early stopping callback

Opened this issue · 0 comments

Current state

Main training function fold_train_ensemble uses early stopping by default: if a number of subepochs are passed (patience) without an improvement in the validation loss, then training stops.

Problem

Sometime the validation loss will move to a plateau with a shallow slope; the validation loss continues to decrease, but at rate which minimal impact on performance.

Suggestion

A new callback is introduced to to stop training if the validation loss doesn't decrease by a certain amount (or fraction) after a certain number of epochs. This should ideally automatically scale to the typical loss values for each particular training and should accurately detect plateaus without fine tuning by the user. It could, for instance, monitor the rate of change of loss. MetricLogger already does something similar by monitoring the loss velocity. It should also be resistant to fluctuations in the loss, which may occur, particularity in heavily weighted data.