How to use multiple GPUs for inference?

Question

How to use multiple GPUs for inference?

xjtuwh opened this issue 2 years ago · 6 comments

I try to test a sequence with 399 frames by demo.py, but there is an error like follows
RuntimeError: CUDA out of memory. Tried to allocate 23.72 GiB (GPU 0; 23.70 GiB total capacity; 595.83 MiB already allocated; 20.43 GiB free; 1.63 GiB reserved in total by PyTorch)
I want to know how can I use multi-GPUs？Thanks.

Answer 1 · 2022-10-05T09:02:18.000Z

Hi @xjtuwh ,
could you please tell me the terminal command that you used?

Answer 2 · 2022-10-06T06:59:22.000Z

python demo_vita/demo.py --config-file configs/coco/vita_ir_R50_bs16_50ep.yaml --input /media/wuhan/disk1/dataset/DSAT/data1/*.bmp --output DSAT_output/data1 --save-frames True --opts MODEL.WEIGHTS output/model_final.pth

Answer 3 · 2022-10-06T07:03:54.000Z

There are 399 images in input ditectory. The model_final,pth is pretained on our custom dataset. We only train our model on images like coco. I find that the images are transformed to 800*800 and I think that this cause out of memory on a single GPU.

Answer 4 · 2022-10-06T07:05:19.000Z

I try to use multiple GPUs by modify # demo = VisualizationDemo(cfg, parallel=True, conf_thres=args.confidence_threshold), but this is useless.

Answer 5 · 2022-10-07T00:34:10.000Z

Could you try the below two?

set MODEL.VITA.ENC_WINDOW_SIZE to 6
since the resolution is high, decrease MODEL.VITA.TEST_RUN_CHUNK_SIZE such as 6

Answer 6 · 2022-10-07T02:07:36.000Z

Ok. thank you very much.