clin1223/VLDet

Memory Leak problem.

Closed this issue · 2 comments

muncok commented

Hi,

I am using pytorch==1.10.0 and A100x8 or V100x8 and VLDet_LbaseCCcap_CLIP_R5021k_640b64_2x_ft4x_caption.yaml config.

The GPU memory usage keeps on increasing after each iteration.

increasing speed is about 8~10 MB / iter.

As I figured out, the increment occurs at

VLDet/train_net.py

Line 169 in c3b9bc5

scaler.scale(losses).backward()

and

Process is finally killed at

VLDet/vldet/modeling/roi_heads/vldet_fast_rcnn.py

Line 422 in c3b9bc5

word_features = torch.zeros_like(zs_weight)

Do yu have any idea on this?

clin1223 commented

Hi,

Yes, I meet the same problem under the pytorch 1.10.0 CUDA 11.1. Just try pytorch 1.9.0. Thanks!

--Chuang

  • 👍1
muncok commented

Hi,

Yes, I meet the same problem under the pytorch 1.10.0 CUDA 11.1. Just try pytorch 1.9.0. Thanks!

--Chuang

After downgrading to 1.9.0, the memory leak disappeared.

Thank you!