CUDA out of memory

Question

CUDA out of memory

Opened this issue 5 months ago · 6 comments

houchenfeng commented 5 months ago

Impressive work!

While running the training code, I encountered a CUDA out of memory error.

Could you please advise on any settings that could reduce the memory requirements?

Answer 1 · 2024-05-28T12:03:37.000Z

You can reduce the batch_size, default is 48.

Answer 2 · 2024-06-04T06:40:50.000Z

I'm having the same issue, where the GPU memory keeps increasing until it exceeds it while training.

Answer 3 · 2024-06-04T07:59:26.000Z

me too. During the training process, the GPU memory will become higher and higher, and no matter what batchsize. Is there too much useless information during training that is not cleaned up in time?

Answer 4 · 2024-06-04T11:36:19.000Z

Hi, I just run a quick test on a RTX 4090 (24GB) GPU. However, during the whole training session, I didn't encounter the problem you mentioned. The original work was trained on One A100 GPU. During the whole session, the monitored memory consumption is around 27%-37% of the total GPU memory. Here is the hardware which I just have tested on:

Cuda 11.3
GPU: RTX 4090(24GB) * 1
CPU: 12 vCPU Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz
Memory: 90GB

Hope it can helps.

Answer 5 · 2024-06-14T13:07:56.000Z

@flww213 @leiershuai @ohhhh2022 I also see the increasing VRAM during training... do you figure out the reason?

Answer 6 · 2024-07-23T08:11:59.000Z

the cost of graphic processer memory gets higher and higher, and finally my 8gb 4060 laptop stopped at gen 129/199 QAQ
then I tried to use cloud gpu for training, without shared gpu memory,3090 24gb died at gen 135/199, only a tiny progress than my 4060 laptop with 8gb+16gb (shared memory)