Error of torchac.

Question

Error of torchac.

wensihan opened this issue 5 years ago · 27 comments

Hello, Fabian:

When I try to install the torchac, I meet the following errors:

(pytorch) [wency@localhost torchac]$ COMPILE_CUDA=no python setup.py
Compiling, cuda_support=False
/home/wency/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'extra_compile_args'
warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: no commands supplied

My environment is gcc 4.8.5, nvcc 8.0, python 3.7, pytorch 1.0.

Answer 1 · 2019-07-13T05:55:33.000Z

This is weird! Did you try removing the extra_compile_args argument in setup.py L67? Which platform do you use?

Answer 2 · 2019-07-17T02:11:36.000Z

My platform is Linux.
I have tried to remove the extra_compile_args argument in setup.py L67. The error still exist.
(pytorch) [wency@localhost torchac]$ COMPILE_CUDA=no python setup.py
Compiling, cuda_support=False
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: no commands supplied

Answer 3 · 2019-07-17T06:29:19.000Z

Does it work with COMPILE_CUDA=no python setup.py compile. Maybe I made a mistake in the README.

Answer 4 · 2019-07-17T06:57:41.000Z

It seems that it doesn't work...
(pytorch) [wency@localhost torchac]$ COMPILE_CUDA=no python setup.py compile
Compiling, cuda_support=False
/home/wency/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'extra_compile_args'
warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: invalid command 'compile'

Answer 5 · 2019-07-18T12:22:44.000Z

Sorry, I meant COMPILE_CUDA=no python setup.py install. does that work?

Answer 6 · 2019-07-22T07:45:07.000Z

(pytorch) [wency@localhost torchac]$ COMPILE_CUDA=no python setup.py install
Compiling, cuda_support=False
/home/wency/anaconda3/envs/pytorch/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'extra_compile_args'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
creating torchac_backend_cpu.egg-info
writing torchac_backend_cpu.egg-info/PKG-INFO
writing dependency_links to torchac_backend_cpu.egg-info/dependency_links.txt
writing top-level names to torchac_backend_cpu.egg-info/top_level.txt
writing manifest file 'torchac_backend_cpu.egg-info/SOURCES.txt'
reading manifest file 'torchac_backend_cpu.egg-info/SOURCES.txt'
writing manifest file 'torchac_backend_cpu.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
/home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/utils/cpp_extension.py:191: UserWarning:

                           !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                          !! WARNING !!

warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
building 'torchac_backend_cpu' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/home
creating build/temp.linux-x86_64-3.7/home/wency
creating build/temp.linux-x86_64-3.7/home/wency/L3C-PyTorch
creating build/temp.linux-x86_64-3.7/home/wency/L3C-PyTorch/src
creating build/temp.linux-x86_64-3.7/home/wency/L3C-PyTorch/src/torchac
creating build/temp.linux-x86_64-3.7/home/wency/L3C-PyTorch/src/torchac/torchac_backend
gcc -pthread -B /home/wency/anaconda3/envs/pytorch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/include -I/home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/include/TH -I/home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/include/THC -I/home/wency/anaconda3/envs/pytorch/include/python3.7m -c /home/wency/L3C-PyTorch/src/torchac/torchac_backend/torchac.cpp -o build/temp.linux-x86_64-3.7/home/wency/L3C-PyTorch/src/torchac/torchac_backend/torchac.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=torchac_backend_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option \u2018-Wstrict-prototypes\u2019 is valid for C/ObjC but not for C++ [enabled by default]
creating build/lib.linux-x86_64-3.7
g++ -pthread -shared -B /home/wency/anaconda3/envs/pytorch/compiler_compat -L/home/wency/anaconda3/envs/pytorch/lib -Wl,-rpath=/home/wency/anaconda3/envs/pytorch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/home/wency/L3C-PyTorch/src/torchac/torchac_backend/torchac.o -o build/lib.linux-x86_64-3.7/torchac_backend_cpu.cpython-37m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.7/torchac_backend_cpu.cpython-37m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for torchac_backend_cpu.cpython-37m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/torchac_backend_cpu.py to torchac_backend_cpu.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
pycache.torchac_backend_cpu.cpython-37: module references file
creating dist
creating 'dist/torchac_backend_cpu-1.0.0-py3.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing torchac_backend_cpu-1.0.0-py3.7-linux-x86_64.egg
creating /home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torchac_backend_cpu-1.0.0-py3.7-linux-x86_64.egg
Extracting torchac_backend_cpu-1.0.0-py3.7-linux-x86_64.egg to /home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages
Adding torchac-backend-cpu 1.0.0 to easy-install.pth file

Installed /home/wency/anaconda3/envs/pytorch/lib/python3.7/site-packages/torchac_backend_cpu-1.0.0-py3.7-linux-x86_64.egg
Processing dependencies for torchac-backend-cpu==1.0.0
Finished processing dependencies for torchac-backend-cpu==1.0.0

With the command, it seems that I have install the torchac. While when I test with the following command, it turn out to be wrong...

(pytorch) [wency@localhost torchac]$ python -c 'import torchac'
*** ERROR: torchac_backend not found. Please see README.

It seems I am failed again.

Answer 7 · 2019-07-26T13:42:40.000Z

(Please use backticks for the future to make rading easer, see formatting.)

The problem now is likely

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++ 4.8.5) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Do you have a newer GCC available on your system?

Answer 8 · 2019-07-26T14:01:31.000Z

If you have it somewhere, you can prepend to path. E.g. on my system I do

export PATH=~/compilers/gcc/gcc-5.5.0/bin:$PATH
export COMPILE_CUDA=auto
python setup.py install;

Answer 9 · 2019-07-31T11:59:18.000Z

Probably, the problem discussed in #4 is also relevant here. Can you try the solution posted there?

Answer 10 · 2019-08-02T12:58:10.000Z

I'm closing this now, re-open if there are still problems.

Answer 11 · 2020-09-22T08:39:22.000Z

Hi!@fab-jul I met the same problems! *** ERROR: torchac_backend not found. Please see README.
I tried the gcc-5 and it failed! And solution in #4 also does not work!

Answer 12 · 2020-09-22T08:46:17.000Z

Hi, which command did you try to compile and what is the exact error message?

Answer 13 · 2020-09-22T09:06:26.000Z

COMPILE_CUDA=auto python setup.py install for cpu
and COMPILE_CUDA=force python setup.py install for gpu

error:
when i run l3c.py, it has:
*** ERROR: torchac_backend not found. Please see README.
and when i import torchac_backend_cpu in python env, it has:
ImportError:xxxx.so, undefined symbol:_ZN2at19undefinedTensorImpl10_singletonE

Answer 14 · 2020-09-22T09:20:39.000Z

And what about the compile command? What‘s the output there?

Answer 15 · 2020-09-22T09:20:55.000Z

Also, which version of pytorch do you have?

Answer 16 · 2020-09-22T09:56:00.000Z

sorry for bothering u again! my pytorch is 1.0.1!
compile command is COMPILE_CUDA=auto python setup.py install according to readme! and output is:
`COMPILE_CUDA=auto python setup.py install
Found CUDA supported: False
Compiling, cuda_support=False
/home/xxx/anaconda3/envs/py36/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'extra_compile_args'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
creating torchac_backend_cpu.egg-info
writing torchac_backend_cpu.egg-info/PKG-INFO
writing dependency_links to torchac_backend_cpu.egg-info/dependency_links.txt
writing top-level names to torchac_backend_cpu.egg-info/top_level.txt
writing manifest file 'torchac_backend_cpu.egg-info/SOURCES.txt'
reading manifest file 'torchac_backend_cpu.egg-info/SOURCES.txt'
writing manifest file 'torchac_backend_cpu.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'torchac_backend_cpu' extension
gcc -pthread -B /home/xxx/anaconda3/envs/py36/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages/torch/lib/include/THC -I/home/xxx/anaconda3/envs/py36/include/python3.6m -c /home/xxx/Desktop/compression-master/L3C-PyTorch-master/src/torchac/torchac_backend/torchac.cpp -o build/temp.linux-x86_64-3.6/home/xxx/Desktop/compression-master/L3C-PyTorch-master/src/torchac/torchac_backend/torchac.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=torchac_backend_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /home/xxx/anaconda3/envs/py36/compiler_compat -L/home/xxx/anaconda3/envs/py36/lib -Wl,-rpath=/home/xxx/anaconda3/envs/py36/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/home/xxx/Desktop/compression-master/L3C-PyTorch-master/src/torchac/torchac_backend/torchac.o -o build/lib.linux-x86_64-3.6/torchac_backend_cpu.cpython-36m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/torchac_backend_cpu.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/torchac_backend_gpu.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for torchac_backend_cpu.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/torchac_backend_cpu.py to torchac_backend_cpu.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torchac_backend_cpu.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
pycache.torchac_backend_cpu.cpython-36: module references file
creating 'dist/torchac_backend_cpu-1.0.0-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing torchac_backend_cpu-1.0.0-py3.6-linux-x86_64.egg
creating /home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages/torchac_backend_cpu-1.0.0-py3.6-linux-x86_64.egg
Extracting torchac_backend_cpu-1.0.0-py3.6-linux-x86_64.egg to /home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages
Adding torchac-backend-cpu 1.0.0 to easy-install.pth file

Installed /home/xxx/anaconda3/envs/py36/lib/python3.6/site-packages/torchac_backend_cpu-1.0.0-py3.6-linux-x86_64.egg
Processing dependencies for torchac-backend-cpu==1.0.0
Finished processing dependencies for torchac-backend-cpu==1.0.0
`

Answer 17 · 2020-09-22T11:33:37.000Z

So to me the previous error sounds like it could be related to your pytorch version. Could you try with 1.1? See README.

Answer 18 · 2020-09-22T11:38:41.000Z

Oh!, Maybe i found the reason, nvcc is 5.5 not 9.

Answer 19 · 2020-09-22T11:46:43.000Z

Could also be. You can also try with COMPILE_CUDA=no iirc

But using pytorch 1.1 is also important I think.

Answer 20 · 2020-09-23T09:33:58.000Z

Did you get it to run?

Answer 21 · 2020-09-23T10:24:27.000Z

:(
I use pytorch1.1 and it does not work. nvcc=9 is also not right.
But i do not know what is iirc

You can also try with COMPILE_CUDA=no iirc

Answer 22 · 2020-09-24T08:51:58.000Z

iirc == if I recall correctly

what I meant is you can try to compile with

COMPILE_CUDA=no python setup.py install

Answer 23 · 2020-09-24T08:53:27.000Z

What does pip list show? And I think the compilation is actually fine, the problem seems to be the

ImportError:xxxx.so, undefined symbol:_ZN2at19undefinedTensorImpl10_singletonE

which sounds like there is some incorrect static library somewhere. not sure which or what. you could also try the fix used in the pull request #19.

Answer 24 · 2020-09-25T02:46:28.000Z

pip list is normal. So i think compilation is also fine. And COMPILE_CUDA=no can't help me.

All in all, i will try the solution in the pull request #19.

Answer 25 · 2020-09-25T08:19:47.000Z

Ok, keep me posted!

Answer 26 · 2020-10-17T08:40:55.000Z

Thanks God! I finally solve this problem with #https://github.com/caoscott/SReC/blob/master/INSTALL.md.

Answer 27 · 2020-11-14T11:27:10.000Z

FYI, I released this as a standalone library that does not need CUDA here: https://github.com/fab-jul/torchac