The training hangs after reloading one of master/worker pods

Question

The training hangs after reloading one of master/worker pods

dmitsf opened this issue 3 years ago · 5 comments

Hello!
I'm setting up training with PyTorchJobs. I have the problem: if one of the pods (doesn't matter, master or worker) reloads, the whole process hangs. The reason for reloading can be different, usually, it's due to Google Cloud Engine node rescheduling. Also, I tried to kill pods myself - the behavior was the same.
Can I avoid this behavior and make training tolerant to pods' reloading?

Answer 1 · 2021-10-29T02:31:58.000Z

Can you tell us the pytorch version?

Answer 2 · 2021-10-29T12:27:11.000Z

I use pytorch 1.9.0.

Answer 3 · 2021-10-29T14:13:24.000Z

Are you using torch.distributed.run?

Answer 4 · 2021-10-29T16:34:38.000Z

I don't use it at the moment.
I followed mnist example to adjust my training script.

Answer 5 · 2021-10-30T01:49:30.000Z

Can you please show us the script and the YAML file? PyTorch 1.9 introduced elastic training and it may hang.