CUDA out of memory with 24gb VRAM
Opened this issue · 3 comments
Im getting a CUDA out of memory error with continual inpainting and segmenting. seems to build up VRAM usage and not release it. here it is stuck at 1/20 and continually eating up more ram, over 50gb.
I was unable to reproduce the issue of increasing VRAM usage when repeatedly using SAM and Inpainting on Linux.
Did you try swapping models frequently? I left that bit out. it might not be unloading models correctly, say swapping segmenting models then swapping the inpaint model a few times. I've recreated it on two computers, a RTX2080ti, and my main PC, RTX 4090, both running windows 11, CUDA 12.4, latest nvidia drivers. no other issues in comfyui or a1111 inpainting with the main interface and swapping models as normal
I monitored the VRAM usage with nvidia-smi -l and found that VRAM is properly released after SAM and Inpainting operations conclude, which seems to prevent the issue from being reproduced on Linux.
I will continue to investigate this issue in a Windows environment as well.