Stoping and restarting training script starts from scratch
turowicz opened this issue · 3 comments
turowicz commented
Hey,
I'm having some serious issues each time I stop the training after a checkpoint has been created and run evaluation.
It seems like the restarted job picks up the checkpoint step number but starts learning from scratch.
Cheers
sglvladi commented
@turowicz, this certainly is odd behaviour.. Can you please specify the commands you used for training and evaluation? A copy of your config file would also be helpfull
turowicz commented
More details here: