subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
SupriyaB1 opened this issue · 7 comments
ros@rosmachine:~/Downloads/pvcnn$ python3 train.py configs/kitti/frustum/pvcnne.py --devices 0 --evaluate --configs.evaluate.best_checkpoint_path /home/ros/Downloads/kitti.frustum.pvcnne.pth.tar --configs.evaluate.num_tests 20
==> loading configs from ['configs/kitti/frustum/pvcnne.py']
Traceback (most recent call last):
File "/home/ros/.local/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1400, in _run_ninja_build
check=True)
File "/usr/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 266, in
main()
File "train.py", line 79, in main
configs = prepare()
File "train.py", line 24, in prepare
configs.update_from_modules(*args.configs)
File "/home/ros/Downloads/pvcnn/utils/config.py", line 113, in update_from_modules
importlib.import_module(module)
File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 953, in _find_and_load_unlocked
File "", line 219, in _call_with_frames_removed
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/ros/Downloads/pvcnn/configs/kitti/frustum/init.py", line 7, in
from meters.kitti import MeterFrustumKitti
File "/home/ros/Downloads/pvcnn/meters/kitti/init.py", line 1, in
from meters.kitti.frustum import MeterFrustumKitti
File "/home/ros/Downloads/pvcnn/meters/kitti/frustum.py", line 4, in
from modules.frustum import get_box_corners_3d
File "/home/ros/Downloads/pvcnn/modules/init.py", line 1, in
from modules.ball_query import BallQuery
File "/home/ros/Downloads/pvcnn/modules/ball_query.py", line 4, in
import modules.functional as F
File "/home/ros/Downloads/pvcnn/modules/functional/init.py", line 1, in
from modules.functional.ball_query import ball_query
File "/home/ros/Downloads/pvcnn/modules/functional/ball_query.py", line 3, in
from modules.functional.backend import _backend
File "/home/ros/Downloads/pvcnn/modules/functional/backend.py", line 8, in
sources=[os.path.join(src_path,'src', f) for f in [
File "/home/ros/.local/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 898, in load
is_python_module)
File "/home/ros/.local/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1086, in jit_compile
with_cuda=with_cuda)
File "/home/ros/.local/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1186, in write_ninja_file_and_build_library
error_prefix="Error building extension '{}'".format(name))
File "/home/ros/.local/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1413, in run_ninja_build
raise RuntimeError(message)
RuntimeError: Error building extension 'pvcnn_backend': [1/14] c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/bindings.cpp -o bindings.o
FAILED: bindings.o
c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/bindings.cpp -o bindings.o
In file included from /home/ros/.local/lib/python3.7/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/pybind11/cast.h:13,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/pybind11/attr.h:13,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/ros/Downloads/pvcnn/modules/functional/src/bindings.cpp:1:
/home/ros/.local/lib/python3.7/site-packages/torch/include/pybind11/detail/common.h:112:20: fatal error: Python.h: No such file or directory
compilation terminated.
[2/14] c++ -MMD -MF trilinear_devox.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/trilinear_devox.cpp -o trilinear_devox.o
FAILED: trilinear_devox.o
c++ -MMD -MF trilinear_devox.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/trilinear_devox.cpp -o trilinear_devox.o
In file included from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/Device.h:3:0,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/extension.h:6,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:6,
from /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/trilinear_devox.hpp:4,
from /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/trilinear_devox.cpp:1:
/home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/python_headers.h:9:20: fatal error: Python.h: No such file or directory
compilation terminated.
[3/14] c++ -MMD -MF neighbor_interpolate.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/neighbor_interpolate.cpp -o neighbor_interpolate.o
FAILED: neighbor_interpolate.o
c++ -MMD -MF neighbor_interpolate.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/neighbor_interpolate.cpp -o neighbor_interpolate.o
In file included from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/Device.h:3:0,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/extension.h:6,
from /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/neighbor_interpolate.hpp:4,
from /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/neighbor_interpolate.cpp:1:
/home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/python_headers.h:9:20: fatal error: Python.h: No such file or directory
compilation terminated.
[4/14] c++ -MMD -MF grouping.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/grouping/grouping.cpp -o grouping.o
FAILED: grouping.o
c++ -MMD -MF grouping.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/grouping/grouping.cpp -o grouping.o
In file included from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/Device.h:3:0,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/extension.h:6,
from /home/ros/Downloads/pvcnn/modules/functional/src/grouping/grouping.hpp:4,
from /home/ros/Downloads/pvcnn/modules/functional/src/grouping/grouping.cpp:1:
/home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/python_headers.h:9:20: fatal error: Python.h: No such file or directory
compilation terminated.
[5/14] c++ -MMD -MF ball_query.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/ball_query/ball_query.cpp -o ball_query.o
FAILED: ball_query.o
c++ -MMD -MF ball_query.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++17 -c /home/ros/Downloads/pvcnn/modules/functional/src/ball_query/ball_query.cpp -o ball_query.o
In file included from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/Device.h:3:0,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
from /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/extension.h:6,
from /home/ros/Downloads/pvcnn/modules/functional/src/ball_query/ball_query.hpp:4,
from /home/ros/Downloads/pvcnn/modules/functional/src/ball_query/ball_query.cpp:1:
/home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/python_headers.h:9:20: fatal error: Python.h: No such file or directory
compilation terminated.
[6/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_37,code=sm_37 --compiler-options '-fPIC' -std=c++14 -c /home/ros/Downloads/pvcnn/modules/functional/src/ball_query/ball_query.cu -o ball_query.cuda.o
[7/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_37,code=sm_37 --compiler-options '-fPIC' -std=c++14 -c /home/ros/Downloads/pvcnn/modules/functional/src/interpolate/neighbor_interpolate.cu -o neighbor_interpolate.cuda.o
[8/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/TH -isystem /home/ros/.local/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_37,code=sm_37 --compiler-options '-fPIC' -std=c++14 -c /home/ros/Downloads/pvcnn/modules/functional/src/grouping/grouping.cu -o grouping.cuda.o
ninja: build stopped: subcommand failed.
i get the same error while running on cuda-11, but when I tried the model on colab, it runs fine
It seems that the error is not from our codebase. It's triggered by torch/include/torch/csrc/python_headers.h:9:20, and the error is due to the missing Python.h header (see https://stackoverflow.com/questions/21530577/fatal-error-python-h-no-such-file-or-directory).
I'm closing this issue due to inactivity. Please feel free to reopen it if the problem has not been resolved.
it's resolved now. It was happening because of the improper cudnn version
it's resolved now. It was happening because of the improper cudnn version
what's your cudnn version now?
I used this container: nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04 and no error then
Is it possible to use in newer cuda version?