Maybe a small bug about test progress bar in multi_gpu_test().
MisakaMikt opened this issue · 1 comments
During distributed training, sot model use multi_gpu_test() func to test all videos on several GPUs. But the progress bar is updated on rank 0.
The number of frames on each rank is not equal, in my run, the number of frames on rank0 is less than total_frames / world_size, so the progress bar will be less than the actual total number of frames. For example, I used 3 gpu, and total 21007 frames(got10k val), and here is some out info. the final progress only 5565*3=16695.
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ] 16695/21007, 99.7 task/s, elapsed: 167s, ETA: 43s
rank 0 world_size 3 cnt 5565.
rank 1 world_size 3 cnt 7072.
rank 2 world_size 3 cnt 8370.
Hello @MisakaMikt
Haven't you got this MMCV compatibility error?