StructureEncoder can occupy 2G GPU memory and cost another 2G GPU memory when do inference. whether can I reduce the GPU usage?

Question

StructureEncoder can occupy 2G GPU memory and cost another 2G GPU memory when do inference. whether can I reduce the GPU usage?

Closed this issue a year ago · 4 comments

Thanks for your awesome work!

Answer 1 · 2024-01-02T08:28:39.000Z

Hi, you cann try to inference with datatype torch.float16. Convert the model and input into float16, then forward the data with the model, and convert the output back into float32.

Answer 2 · 2024-01-02T08:47:49.000Z

Hi, you cann try to inference with datatype torch.float16. Convert the model and input into float16, then forward the data with the model, and convert the output back into float32.

Thx for your reply and kind advice. I tried to inference with fp16(FFT op use float32). Indeed it can reduce the gpu memory usage. But I can't evaluate the difference between fp32 and fp16 even though a couple of image getting same results. Have you tried some inference framework that can help reduce the gpu memory?

Answer 3 · 2024-01-02T08:53:21.000Z

You can evalute the images from Places2 test split with/without fp16, then evalute the inpaining results with PSNR/SSIM/FID/LPIPS quantitatively.

For the second question, I did't try other methods for memroy reduction.

Answer 4 · 2024-01-03T09:37:04.000Z

You can evalute the images from Places2 test split with/without fp16, then evalute the inpaining results with PSNR/SSIM/FID/LPIPS quantitatively.

For the second question, I did't try other methods for memroy reduction.

3Q for your constructive feedback. I tried to convert model to onnx but failed due to ops related to FFT and find other method for memory reduction. Thanks again!