uci-rendering/psdr-cuda

Memory leak when optimization

Closed this issue · 8 comments

When I try to optimize the vertices positions using the provided optimization code here #11. The GPU memory usages increases along iterations and eventually out of memory. How can I avoid memory leak?

you will need to clear the grad(like zerograd) / delete loss, or call torch.cuda.empty_cache() or ek.cuda_malloc_trim() every time you finish the opt.step()

I have deleted all the variable generated by enoki in backward operation and call ek.cuda_malloc_trim every step. However, the occupied video memory fluctuates up and down but still gradually increases. Is there any clue?

Sound pretty strange, I can take a look on your code if needs

Here is my optimization code
code.zip. At the beginning, 3G gpu memory is required. As the number of iterations increased, the gpu memory usage fluctuates up and down but gradually increases to 5g.

It seems that,if you configure the scene,by scene.configure ever iteration,it will up load geometry to gpu which increase the memory graudually.

Please do the following update and see if the problem gone. Sorry for the late update

I have done the update as you said, and there is no memory leak now. Thank you very much!