memory leak in the training code

Question

memory leak in the training code

Closed this issue 10 months ago · 4 comments

There is a memory leak in the training code of this grnet model, and my training memory of more than 800epoch can occupy about 40G.

Answer 1 · 2023-08-20T04:24:52.000Z

Sorry, I cannot reproduce your issue.
At the time I conducted this experiment, I only had TITAN Xp with 12GB memory. It is impossible to occupy 40 GB of memory.

Answer 2 · 2023-08-23T01:40:52.000Z

The training data I use is Completion3D.

Answer 3 · 2023-08-23T01:56:54.000Z

By the memory, I don't mean the graphics memory of the graphics card, but the memory of the computer.

Answer 4 · 2023-08-23T02:01:31.000Z

Throughout the training process, all operations are executed on CUDA. If there is indeed a memory leak issue, my suspicion is that it might be associated with PyTorch or other Python libraries.