Cupy: cudaErrorIllegalAddress: an illegal memory access was encountered
Closed this issue · 5 comments
Hi,
thanks for your great work. I run into the above mentioned cuda error when running the cupy script on the following normal map.
I was wondering whether you can reproduce the issue and have an idea where to start looking. Potentially somewhere where the masks are computed or the diagonal_data_term is updated.
It works with the numpy script, but I want to use the cupy script as it is faster for many of my normal_maps.
Looking forward to hearing from you.
Best,
Jan
Hi, I have tried running the cupy script on your data, but I did not encounter any issue. I tested using cuda 11.3 with cupy-cuda11x.
I guess this cuda error are possibly due to the conflict with other threads using cuda (e.g., pytorch) or parallelization running.
Have you tried running the script along?
Hope it helps.
Hi,
thanks for your quick response and testing it on my data.
For me the error is reproducible for a number of shapes. I could not figure out yet what they have in common. Independent of the time or machine that I run on. Also there are no other GPU processes running when I start the script.
These are my cuda, cupy versions:
- cuda/10.2.89
- cudnn/10.2_v7.6
- cupy-cuda102 (name) 11.3.0 (version)
So I'll try with a newer cuda version. Thanks for the hint.
How about commenting out the two lines?
https://github.com/hoshino042/bilateral_normal_integration/blob/a224fb3897ac7c3a59f7e2c97d497962febd6604/bilateral_normal_integration_cupy.py#L16
The two lines are used to avoid out-of-memory for huge normal maps (like >2k). But in your case, the normal maps are several hundreds pixels wide. It is then safe to ignore such explicit memory management.
I tried it with the two lines commented out but the error remains.
I also tried with rather large images with 5000x3000 pixels but that worked. Even with the cuda memory allocator in place.
I updated my environment to use cuda/11.3 and respectively cupy-cuda11x (name) 11.4.0 (version)
Additional remark: I had to update pyvista to 0.37.0 to resolve another error
(ImportError: cannot import name 'vtkExtractEdges' from 'vtkmodules.vtkFiltersExtraction')
This resolves my issue. Thanks again for the hint.