dist_train keep waiting

Question

dist_train keep waiting

FX-STAR opened this issue 3 years ago · 4 comments

My env：
cuda10.2
torch==1.6.0
mmdetection==2.8.0
mmcv==1.2.4
After some iters the GPU-Util 100% but the process is always waiting
Could you provide your env or any advice?

Answer 1 · 2021-01-27T00:56:02.000Z

Hi, I didn't run into this problem so can't provide effective solutions. The code is tested with: cuda 10.1, pytorch 1.6.0, mmdet 2.5.0, and mmcv=1.1.5. You may have a look at this page for more information about training.

Answer 2 · 2021-03-06T11:23:48.000Z

@whoNamedCody ,hi,has the problem been resolved? I also faced this problem.

Answer 3 · 2021-03-09T01:13:54.000Z

@whoNamedCody ,hi,has the problem been resolved? I also faced this problem.

no, i think the problem maybe in 'GiouLoss', but not debug yet

Answer 4 · 2021-03-11T07:06:05.000Z

hi, Thank you very much for your reply! The problem has been solved. At 2021-03-09 09:14:10, "FuxingLeng" <notifications@github.com> wrote: @whoNamedCody ,hi,has the problem been resolved? I also faced this problem. no, i think the problem maybe in 'GiouLoss', but not debug yet — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.