CUDA out of memory，even set batch_size to 1
Opened this issue · 8 comments
RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 8.00 GiB total capacity; 5.69 GiB already allocated; 4.04 MiB free; 5.88 GiB reserved in total by PyTorch)
i have set batch_size to 1, but it still occur oom
What sort of GPU do you have? It seems the most likely explanation is that you have another process that is using the GPU memory. If you set "fp16_run": true in https://github.com/NVIDIA/waveglow/blob/master/config.json this also reduces memory use.
I also have this question, my GPU is nvidia P100, and I'm sure that there is no other process that is using the GPU. I think it is because the model is too big, so I use apex.amp to try to train the model and it works.
I can run with batch_size=1 on 1050ti 4GB, you should check your system for running processes
You can decrease the segment length if running out of memory. You should be able to fit batch size 1 on a 8GB GPU.
Closing due to inactivity.
You can decrease the segment length if running out of memory. You should be able to fit batch size 1 on a 8GB GPU.
Closing due to inactivity.
in case I have plenty of VRAM left, should I increase segment length?
how does segment length affect accuracy, inference speed, training speed?
We haven't explored the effect of changing segment length on output quality.
Shorter segment lengths will result in more padding due to the large receptive field. This is not helpful for optimization.
Was this ever resolved. I'm getting this on a RTX 2070 Super 8GB. I've tried 1 BS and 1,000 segment size and still get the same error. I'm also seeing the following warning...Could this be related. I found I needed to upgrade the NGC docker version of PyTorch to be compatible with my Cuda version. Should we be using a more recent NGC?
Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ImportError('/opt/conda/lib/python3.6/site-packages/amp_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIdEEPKNS_6detail12TypeMetaDataEv',)
In Google Colab, with a batch size of 1, it gives out of memory error for an audio 5 seconds long.
waveglow = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_waveglow', model_math='fp32')
waveglow = waveglow.remove_weightnorm(waveglow)
waveglow = waveglow.to('cuda')
waveglow.eval()
audio = waveglow.infer(mel.cuda())
Before the model => GPU Memory = 1100MiB / 15109MiB
After defining the model => GPU Memory = 15046MiB / 15109MiB