LarsKue/lightning-trainable

Progressbar confusing when continuing training of a loaded model.

Closed this issue · 1 comments

When continuing to train a model loaded from a checkpoint it will train for {max_epochs - "already trained epochs"}.

Example:

  • Loaded model already trained 200 epochs.
  • max_epochs = 400
  • => training will be 200 epochs

Screenshot from 2023-05-30 10-34-14
The epoch counter on the left shows total trained epochs. The counter on the right shows trained epochs in the current run, but the total on the right is max_epochs although training will only be half of that.

When the training is finished the progressbar is only at the halfway line.
Screenshot from 2023-05-30 11-00-41
(Sorry, different example.)

Reverted to the default lightning progress bar for now.