Theano/libgpuarray

GpuArrayException: Could not load "nvrtc64_90.dll": Le module spécifié est introuvable.

Closed this issue · 11 comments

notoraptor commented

Just to report a very strange issue about libgpuarray on Windows (8.1 64 bits). It now looks for a DLL named nvrtc64_90.dll but it can't find it on my computer:

(python2) HPPC@NOTORAPTOR-HP C:\Users\HPPC\mila\dev\git\theano
> set THEANO_FLAGS=device=cuda

(python2) HPPC@NOTORAPTOR-HP C:\Users\HPPC\mila\dev\git\theano
> python -c "import theano"

Tmp 9000 Major 9 Minor 0
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "theano\gpuarray\__init__.py", line 179, in <module>
    use(config.device)
  File "theano\gpuarray\__init__.py", line 166, in use
    init_dev(device, preallocate=preallocate)
  File "theano\gpuarray\__init__.py", line 65, in init_dev
    sched=config.gpuarray.sched)
  File "pygpu\gpuarray.pyx", line 634, in pygpu.gpuarray.init (pygpu/gpuarray.c:9436)
  File "pygpu\gpuarray.pyx", line 584, in pygpu.gpuarray.pygpu_init (pygpu/gpuarray.c:9127)
  File "pygpu\gpuarray.pyx", line 1057, in pygpu.gpuarray.GpuContext.__cinit__ (pygpu/gpuarray.c:13429)
GpuArrayException: Could not load "nvrtc64_90.dll": Le module spécifié est introuvable.

Looking into NVIDIA bin folder (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin), I found nvrtc64_80.dll but not nvrtc64_90.dll:

image

NB: it seems the problem comes from library version detection, based on the value returned by cuDriverGetVersion():
https://github.com/Theano/libgpuarray/blob/master/src/gpuarray_buffer_cuda.c#L143
https://github.com/Theano/libgpuarray/blob/master/src/loaders/libnvrtc.c#L30

But it worked some days ago. I just reminded that I have updated my NVIDIA driver some days ago too, but it should not cause any problem, should it ?

@abergeron @nouiz @lamblin

abergeron commented

This it because you have a very recent driver that advertises cuda 9.0 as the version. Try setting GPUARRAY_CUDA_VERSION to 80 and it should load the right version.

If you have an idea to improve this, I'm all ears.

  • 👍2
notoraptor commented

Thanks @abergeron , that resolves the problem.

No idea for the moment ! But this means there is not link between driver version retrieved from nvcuda.dll and CUDA version installed by user. Maybe we should just loop over known versions (from the one given by cuDriverGetVersion() (here 90) to the oldest supported (e.g. 75 ?)) until we find the right nvrtc64_<version>.dll ?

abergeron commented

We could do a loop to find the first matching library yes. The only thing we have to make sure is that we load the same version for nvrtc and cublas.

notoraptor commented

OK ! I will do a PR about that this week (or the next one).

I close the issue, as there is a working solution.

nouiz commented
pranitapradhan91 commented

@notoraptor @abergeron

I am running into the same problem as mentioned here. Everything was working fine a week back https://github.com/Theano/Theano/issues/6507#event-1332196252, however I am getting this error since today.

'GpuArrayException: Could not load "nvrtc64_70.dll": The specified module could not be found.'

I am using pygpu 0.7.4, cuda 9 and theano 1.0.0. I did set GPUARRAY_CUDA_VERSION=80, but still it is not working. I found this file residing in the bin folder.

image

Is there anything I can do? Some checks? Or am I doing something wrong?

Waiting for reply.

Thank you in advance.

notoraptor commented

Variable GPUARRAY_CUDA_VERSION should be useless in the latest versions of pygpu. It seems pygpu cannot find your cuda installation.

  • Can you update to pygpu 0.7.5 ?
  • What is the content of your environment variable CUDA_PATH ?
pranitapradhan91 commented

@notoraptor , thank you for your reply as always.
Yes I updated pygpu to 0.7.5 and that worked.

Deleted user commented

I'm currently having the same issue as @pranitapradhan91 had. And my pygpu version is 0.7.5 but its still throwing me the same error.

alejandroesteban commented

@mineo210 Hi, could you solve the problem? I could not resolve all the drivers again ... but after a while my problem reappeared ... maybe when updating the video driver

Deleted user commented

@alejandroesteban I had no luck solving the problem so after a while I decided to give up, after some months I tried again but still did not succeed. So the answer to your answer is sadly a no.

  • 👍2