error when running "from modules.functional.backend import _backend"
Carbord opened this issue · 11 comments
Hello! When running " from modules.functional.backend import _backend ", I encounter the following error :
It may result from the wrong version. Can you provide the versions about the pytorch, nvcc, etc to run this repostory?
I am using pytorch 1.11. And the "ncvv -V" shows :
while nvidia-smi shows :
I wonder how to correctly run the code to import the corresponding functions about pvcnn_backend. Sincerely looking forward to your help.
Could you please post the full error message here? Thanks!
Not the original poster but I had a similar error
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1740, in _run_ninja_build
subprocess.run(
File "/opt/conda/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/jovyan/work/PVD/train_generation.py", line 11, in
from model.pvcnn_generation import PVCNN2Base
File "/home/jovyan/work/PVD/model/pvcnn_generation.py", line 6, in
from modules import SharedMLP, PVConv, PointNetSAModule, PointNetAModule, PointNetFPModule, Attention, Swish
File "/home/jovyan/work/PVD/modules/init.py", line 1, in
from modules.ball_query import BallQuery
File "/home/jovyan/work/PVD/modules/ball_query.py", line 4, in
import modules.functional as F
File "/home/jovyan/work/PVD/modules/functional/init.py", line 1, in
from modules.functional.ball_query import ball_query
File "/home/jovyan/work/PVD/modules/functional/ball_query.py", line 3, in
from modules.functional.backend import _backend
File "/home/jovyan/work/PVD/modules/functional/backend.py", line 6, in
_backend = load(name='pvcnn_backend',
File "/opt/conda/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1144, in load
return jit_compile(
File "/opt/conda/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1357, in jit_compile
write_ninja_file_and_build_library(
File "/opt/conda/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1469, in write_ninja_file_and_build_library
run_ninja_build(
File "/opt/conda/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1756, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'pvcnn_backend': [1/14] c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/bindings.cpp -o bindings.o
FAILED: bindings.o
c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/bindings.cpp -o bindings.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
[2/14] c++ -MMD -MF vox.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/voxelization/vox.cpp -o vox.o
FAILED: vox.o
c++ -MMD -MF vox.o.d -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/voxelization/vox.cpp -o vox.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
[3/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --compiler-bindir=/usr/bin/gcc-8 -std=c++14 -c /home/jovyan/work/PVD/modules/functional/src/sampling/sampling.cu -o sampling.cuda.o
[4/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --compiler-bindir=/usr/bin/gcc-8 -std=c++14 -c /home/jovyan/work/PVD/modules/functional/src/interpolate/trilinear_devox.cu -o trilinear_devox.cuda.o
[5/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --compiler-bindir=/usr/bin/gcc-8 -std=c++14 -c /home/jovyan/work/PVD/modules/functional/src/grouping/grouping.cu -o grouping.cuda.o
[6/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --compiler-bindir=/usr/bin/gcc-8 -std=c++14 -c /home/jovyan/work/PVD/modules/functional/src/ball_query/ball_query.cu -o ball_query.cuda.o
[7/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --compiler-bindir=/usr/bin/gcc-8 -std=c++14 -c /home/jovyan/work/PVD/modules/functional/src/interpolate/neighbor_interpolate.cu -o neighbor_interpolate.cuda.o
[8/14] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' --compiler-bindir=/usr/bin/gcc-8 -std=c++14 -c /home/jovyan/work/PVD/modules/functional/src/voxelization/vox.cu -o vox.cuda.o
[9/14] c++ -MMD -MF trilinear_devox.o.d -DTORCH_EXTENSION_NAME=_pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/interpolate/trilinear_devox.cpp -o trilinear_devox.o
[10/14] c++ -MMD -MF ball_query.o.d -DTORCH_EXTENSION_NAME=_pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/ball_query/ball_query.cpp -o ball_query.o
[11/14] c++ -MMD -MF sampling.o.d -DTORCH_EXTENSION_NAME=_pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/sampling/sampling.cpp -o sampling.o
[12/14] c++ -MMD -MF grouping.o.d -DTORCH_EXTENSION_NAME=_pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/grouping/grouping.cpp -o grouping.o
[13/14] c++ -MMD -MF neighbor_interpolate.o.d -DTORCH_EXTENSION_NAME=_pvcnn_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1014" -isystem /opt/conda/lib/python3.9/site-packages/torch/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -O3 -std=c++17 -c /home/jovyan/work/PVD/modules/functional/src/interpolate/neighbor_interpolate.cpp -o neighbor_interpolate.o
ninja: build stopped: subcommand failed.
Interestingly if I run the code again after it fails the first time, it'll start working.
Here's some additional information:
Dockerfile:
FROM gitlab-registry.nrp-nautilus.io/prp/jupyter-stack/prp:54ef7146
LABEL maintainer="Steven Tsan stsan@ucsd.edu"
USER root
RUN apt-get update &&
apt-get install -y vimRUN pip install jetnet
pip install mplhepRUN echo 'export CUDA_HOME=/usr/local/cuda' >> ~/.bashrc &&
echo 'export PATH=${CUDA_HOME}/bin:${PATH}' >> ~/.bashrc &&
echo 'export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc &&
source ~/.bashrcRUN pip install matplotlib
tqdm
open3d
trimesh
scipy
NinjaRUN apt-get --yes install build-essential
RUN apt-get --yes install g++-8
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_Oct_11_21:27:02_PDT_2021
Cuda compilation tools, release 11.4, V11.4.152
Build cuda_11.4.r11.4/compiler.30521435_0
torch version 1.11.0
nvidia-smi
The error is triggered by c++: fatal error: Killed signal terminated program cc1plus. Could you maybe check if it's due to the memory size?
@zhijian-liu I encounter the question of "No module named '_pvcnn_backend' when I try to run this code. My system is
Ubuntu18.04, Cuda is 10.1, Pytorch is 1.4.0. Also, I tried Ubuntu 18.04 cuda 10.2, pytorch 1.7 still have this problem. The GPU memory is 24G.
@zhijian-liu Thank you for you code and idea, very good.
Could you maybe add verbose=True to
pvcnn/modules/functional/backend.py
Line 6 in db13331
I tried to add the code verbose=True to the backend.py file, however, the error is still the same.
Could you maybe directly import this file and let me know if there is any warning message?
@zhijian-liu When I tried to import these files directly, the error was the same. "ImportError: No module named '_pvcnn_backend'".
Could you do python modules/functional/backend.py?
@zhijian-liu I have already solved this problem, thank you. When I reboot the computer, the code is okay now. Come again, thank you.