golbin/WaveNet

RuntimeError: cuda runtime error (2) : out of memory

Opened this issue · 6 comments

Deleted user commented

Hi! I'm running the following command to train the model:
$ python train.py --data_dir=./test/data --output_dir=./outputs

The GPU I'm using has 16276MiB. However, I get an out of memory error immediately:

/wavenet/networks.py", line 88, in forward
gated = gated_tanh * gated_sigmoid
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCStorage.cu:58

Any thoughts as to why this might be happening? Based on my calculations, the input size is 1x100,000x256 which should easily fit in the 16276 MiB of memory that the GPU has.

sh0416 commented

Me too.

Hyeokreal commented

Hi guys, @angad9 @sh0416. Try code in the pull request. You should change the residual stacking part to avoid gpu out of memory issue.

sh0416 commented

Despite using your pull request, it doesn't work. @Hyeokreal

Abel1802 commented

Despite using your pull request, it doesn't work. @Hyeokreal

Me too

methylpentane commented

me too.
it worked on my GTX1080Ti with stack_size=1 and layer_size=5, and don't worked with more layer_size...
this model include very very deep convolution and input size is big, so perhaps it's natural.

GitHubGeniusOverlord commented

You can reduce the batch size. In config.py, change the sample_size - default to a lower value. This should be enough and you dont need to mess with the rest of the model.

  • 👍1