Stuck when start training

[mc171819]

Hi, I tried to run your training code, however, I cannot start training successfully. I have stucked here for a long time. Do you have advice to start training?

@xinge008

[mc171819]

The CPU occupation is high now but gpu occupation is low. Other project work fine on gpu so it is not my gpu problem. This is the output of nvidia-smi

[mc171819]

after waiting for about half an hour, it start training and all things seems normal. It seems epoch 0 iter0 is not on gpu training. It's an weird problem.

[mc171819]

The problem is the validation of iter 0 ep 0 before training, add global_iter>0 on validation solute it.