cuda runtime error (77) : an illegal memory access was encountered
Closed this issue · 3 comments
Hi,
while training cityscapes, i get the following error radomly during 1st or 2nd epoch:
THCudaCheck FAIL file=/home/udo/programs/torch/extra/cutorch/lib/THC/generic/THCStorage.c line=30 error=77 : an illegal memory access was encountered /home/udo/programs/torch/install/bin/luajit: cuda runtime error (77) : an illegal memory access was encountered at /home/udo/programs/torch/extra/cutorch/lib/THC/generic/THCStorage.c:147
I am using Ubuntu 16.04 with Cuda 8.0 and a GTX1070
Doesn't matter if cudnn is installed via luarocks install or not....
Best Regards,
Udo
Edit: launching with CUDA_LAUNCH_BLOCKING=1, i get:
THCudaCheck FAIL file=/home/udo/programs/torch/extra/cunn/lib/THCUNN/ClassNLLCriterion.cu line=171 error=77 : an illegal memory access was encountered /home/udo/programs/torch/install/bin/luajit: cuda runtime error (77) : an illegal memory access was encountered at /home/udo/programs/torch/extra/cutorch/lib/THC/generic/THCStorage.c:147
@uschmelmer I used to get this error when my image or batch won't fit or 'almost' fill up gpu memory.
Hi, i was due to a corrupt Path to the dataset. The Code did Not give any error until the Crash after some time.
Best Regards,
Udo
I have fixed this problem by install
git clone https://github.com/torch/cunn
cd cunn
luarocks make rocks/cunn-scm-1.rockspec