Kaixhin/dockerfiles

Problem with six version and cuda-theano

fakarthur opened this issue · 3 comments

fakarthur commented

Hi Kaixhin,
first, thanks for the great images! I want to report a problem that I have with the cuda-theano image though. I am still having the issue which was marked as resolved here.

Here are my specs:

  • Ubuntu 14.04
  • NVIDIA driver version: 361.93.02
  • GPU: Quadro K6000

With the images kaixhin/cuda-theano:7.5 and kaixhin/cuda-theano:8.0 I get the following error after importing theano:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/theano/__init__.py", line 124, in <module>
    from theano.scan_module import (scan, map, reduce, foldl, foldr, clone,
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/__init__.py", line 41, in <module>
    from theano.scan_module import scan_opt
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_opt.py", line 71, in <module>
    from theano.scan_module import scan_op
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_op.py", line 58, in <module>
    from six import iteritems, integer_types, raise_from
ImportError: cannot import name raise_from

I have six-1.5.2 installed in /usr/libs/python2.7 and six-1.11.0 in /usr/local/libs/python2.7.

If I enforce python to use the six version in /usr/local/lib/python2.7 (by putting /usr/local/lib/python2.7/dist-packages on the beginning of my sys.path, I get another error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/theano/__init__.py", line 124, in <module>
    from theano.scan_module import (scan, map, reduce, foldl, foldr, clone,
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/__init__.py", line 41, in <module>
    from theano.scan_module import scan_opt
  File "/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_opt.py", line 60, in <module>
    from theano import tensor, scalar
ImportError: cannot import name tensor

If I install six-1.11.0 manually using setup.py from https://pypi.python.org/pypi/six#downloads, it seems to work. At least I get a different error now:

Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
In file included from /tmp/try_flags_ZG8wmH.c:4:0:
/usr/include/cudnn.h:63:26: fatal error: driver_types.h: No such file or directory
 #include "driver_types.h"
                          ^
compilation terminated.

Mapped name None to device cuda: Quadro K6000 (0000:03:00.0)

However, with the kaixhin/theano image (without cuda), I don't get the error and everything works fine.

Any idea what is wrong here?

Thanks!

Kaixhin commented

OK I think I've managed to sort this annoying six problem by just deleting it manually and reinstalling (a58f5cc). I don't have a GPU machine to test with atm, so please try the new build of kaixhin/cuda-theano:7.5 and let me know if that solves the cuDNN problem. I'm now building on top of NVIDIA's official cuDNN images, so I doubt an issue with cuDNN itself.

Kaixhin commented

Managed to test, but the cuDNN problem persists. After a bit of searching it seems like this is a common issue for Theano users, so could you post an issue on their repo and see if they can work out what's wrong?

For what it's worth I did find / -name "*cudnn*" and came up with the following (as perhaps it's some path issue):

./usr/local/lib/python2.7/dist-packages/theano/gpuarray/c_code/cudnn_helper.h
./usr/local/lib/python2.7/dist-packages/theano/gpuarray/cudnn_defs.py
./usr/local/lib/python2.7/dist-packages/theano/gpuarray/cudnn_defs.pyc
./usr/share/doc/libcudnn6
./usr/share/doc/libcudnn6-dev
./usr/share/lintian/overrides/libcudnn6
./usr/share/lintian/overrides/libcudnn6-dev
./usr/lib/x86_64-linux-gnu/libcudnn_static.a
./usr/lib/x86_64-linux-gnu/libcudnn.so.6.0.21
./usr/lib/x86_64-linux-gnu/libcudnn.so
./usr/lib/x86_64-linux-gnu/libcudnn_static_v6.a
./usr/lib/x86_64-linux-gnu/libcudnn.so.6
./usr/include/x86_64-linux-gnu/cudnn_v6.h
./usr/include/cudnn.h
./var/lib/dpkg/info/libcudnn6.postrm
./var/lib/dpkg/info/libcudnn6.md5sums
./var/lib/dpkg/info/libcudnn6-dev.list
./var/lib/dpkg/info/libcudnn6.shlibs
./var/lib/dpkg/info/libcudnn6.list
./var/lib/dpkg/info/libcudnn6-dev.prerm
./var/lib/dpkg/info/libcudnn6.postinst
./var/lib/dpkg/info/libcudnn6-dev.postinst
./var/lib/dpkg/info/libcudnn6-dev.md5sums
./var/lib/dpkg/alternatives/libcudnn
./etc/alternatives/libcudnn_stlib
./etc/alternatives/libcudnn
./etc/alternatives/libcudnn_so
fakarthur commented

Thanks for your quick help, Kaixhin! Indeed the six problem seems to be fixed now!
To resolve the cuDNN problem, I had to add the following lines to ~/.theanorc:

[dnn]
enabled=True
include_path=/usr/local/cuda/include
library_path=/usr/local/cuda/lib64

Best greetings!