How to restart training model

Question

How to restart training model

fangrh opened this issue 9 months ago · 2 comments

Hi,

When I run the example of using ABACUS, everything gone well except the training is canceled by the slurm system due to the time limit.

Because the maximum time of using the GPU accelerate card is limited for one submit in our cluster. I wander if I can restart the training when next submit ?

Answer 1 · 2024-01-09T08:51:54.000Z

Please refer to resume in https://deeph-pack.readthedocs.io/en/latest/keyword/train.html#:~:text=best_state_dict.pkl.-,resume,-%3A%20The%20path%20to.

Answer 2 · 2024-01-09T11:17:57.000Z

OK, it works, thank you very much.