Printing training process suddenly stopped during training

Question

Printing training process suddenly stopped during training

Closed this issue 7 months ago · 6 comments

GitHub-Ju commented 10 months ago

During the training process, the printing training process suddenly stopped. What is the reason?

Answer 1 · 2023-12-10T05:20:11.000Z

Not sure, may be similar to #4 (comment). Just stop and try it again.

Answer 2 · 2023-12-10T05:28:12.000Z

If pin_memory is set to true, the following error will be reported, so I set it to false. Is it related to this?

Answer 3 · 2023-12-10T05:46:38.000Z

不确定，可能类似于 #4（评论）。停下来再试一次。

If pin_memory is set to true, the following error will be reported, so I set it to false. Is it related to this?
Or, how to solve this problem?

Answer 4 · 2023-12-10T05:49:55.000Z

I didn't try setting pin_memory to True. In my case, the training stucking problem occasionally and randomly occurs. Sorry but I think the best pratice to handle this issue would still be retrying.

Answer 5 · 2023-12-10T05:51:10.000Z

I didn't try setting pin_memory to True. In my case, the training stucking problem occasionally and randomly occurs. Sorry but I think the best pratice to handle this issue would still be retrying.

alright, thank you very much!

Answer 6 · 2024-01-15T08:04:53.000Z

@GitHub-Ju Hi, you may try this. #17 (comment)