Error building extension '_prroi_pooling'
setarekhosravi opened this issue · 9 comments
setarekhosravi commented
I have this problem when I run run_video.py or run_webcam.py:
python run_video.py dimp dimp50 /media/strh/MyDrive/Track/OtherTracking/tapnet/tapnet_results/second/InShot_20231113_162218069.mp4
Using /home/strh/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/strh/.cache/torch_extensions/py310_cu117/_prroi_pooling/build.ninja...
Building extension module _prroi_pooling...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF prroi_pooling_gpu.o.d -DTORCH_EXTENSION_NAME=_prroi_pooling -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/TH -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda-11.7/include -isystem /home/strh/anaconda3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu.c -o prroi_pooling_gpu.o
FAILED: prroi_pooling_gpu.o
c++ -MMD -MF prroi_pooling_gpu.o.d -DTORCH_EXTENSION_NAME=_prroi_pooling -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/TH -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda-11.7/include -isystem /home/strh/anaconda3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu.c -o prroi_pooling_gpu.o
/media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu.c:17:10: fatal error: THC/THC.h: No such file or directory
17 | #include <THC/THC.h>
| ^~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/home/strh/anaconda3/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/run_video.py", line 38, in <module>
main()
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/run_video.py", line 34, in main
run_video(args.tracker_name, args.tracker_param,args.videofile, args.optional_box, args.debug, args.save_results)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/run_video.py", line 20, in run_video
tracker.run_video_generic(videofilepath=videofile, optional_box=optional_box, debug=debug, save_results=save_results)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/evaluation/tracker.py", line 395, in run_video_generic
out = tracker.track(frame, info)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/evaluation/multi_object_wrapper.py", line 165, in track
out = self.trackers[obj_id].initialize(image, init_info_split[obj_id])
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/tracker/dimp/dimp.py", line 84, in initialize
self.init_classifier(init_backbone_feat)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/tracker/dimp/dimp.py", line 573, in init_classifier
self.target_filter, _, losses = self.net.classifier.get_filter(x, target_boxes, num_iter=num_iter,
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/models/target_classifier/linear_filter.py", line 94, in get_filter
weights = self.filter_initializer(feat, bb)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/models/target_classifier/initializer.py", line 164, in forward
weights = self.filter_pool(feat, bb)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/models/target_classifier/initializer.py", line 45, in forward
return self.prroi_pool(feat, roi1)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/external/PreciseRoIPooling/pytorch/prroi_pool/prroi_pool.py", line 28, in forward
return prroi_pool2d(features, rois, self.pooled_height, self.pooled_width, self.spatial_scale)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/external/PreciseRoIPooling/pytorch/prroi_pool/functional.py", line 44, in forward
_prroi_pooling = _import_prroi_pooling()
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/external/PreciseRoIPooling/pytorch/prroi_pool/functional.py", line 30, in _import_prroi_pooling
_prroi_pooling = load_extension(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension '_prroi_pooling'
and this is cuda, torch, torchvision and GCC version:
gcc --version
gcc (Ubuntu 13.1.0-8ubuntu1~20.04.2) 13.1.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
nvidia-smi
Tue Nov 14 15:30:04 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:02:00.0 Off | N/A |
| N/A 46C P8 N/A / N/A | 9MiB / 2048MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1136 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1735 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0
torch.__version__
'2.0.1+cu117'
>>> torchvision.__version__
'0.15.2'
please help me to solve this problem.
eliafranc commented
Hey,
I was having similar issues and what worked out for me in the end was to actually clone the PreciseRoIPooling repository into the external directory directly. I downloaded the zip file for the PreciseRoIPooling and unpacked it instead and that was causing me problems. PreciseRoIPooling state the following in their Readme:
Causion: To install the library, please git clone the repository instead of downloading the zip file, since source files inside the folder ./pytorch/prroi_pool/src/ and tensorflow/prroi_pool/src/kernels/external are symbol-linked. Downloading the repository as a zip file will break these symbolic links. Also, there are reports indicating that Windows git versions also breaks the symbol links. See vacancy/PreciseRoIPooling#58.
I guess if one would clone the pytracking repository with something like git clone --recurse-submodules it should be fine.
EDIT: I just found out more: The pytracking submodule point to a older version of the PreciseRoIPooling repository. When clicking on the submodule in /ltr/external/ check out the branch you are on on the repository you get redirected to. When switching to the main branch it becomes obvious that some fixing commits were pushed that are crucial for building.
Hope this helps, cheers
setarekhosravi commented
@eliafranc Thank you, but it didn't work for me.
now the error is a little bit different.
Using /home/strh/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/strh/.cache/torch_extensions/py310_cu117/_prroi_pooling/build.ninja...
Building extension module _prroi_pooling...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/local/cuda-11.7:/bin/nvcc -DTORCH_EXTENSION_NAME=_prroi_pooling -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/TH -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda-11.7:/include -isystem /home/strh/anaconda3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' -std=c++17 -c /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu_impl.cu -o prroi_pooling_gpu_impl.cuda.o
FAILED: prroi_pooling_gpu_impl.cuda.o
/usr/local/cuda-11.7:/bin/nvcc -DTORCH_EXTENSION_NAME=_prroi_pooling -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/TH -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda-11.7:/include -isystem /home/strh/anaconda3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 --compiler-options '-fPIC' -std=c++17 -c /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu_impl.cu -o prroi_pooling_gpu_impl.cuda.o
/bin/sh: 1: /usr/local/cuda-11.7:/bin/nvcc: not found
[2/3] c++ -MMD -MF prroi_pooling_gpu.o.d -DTORCH_EXTENSION_NAME=_prroi_pooling -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/TH -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda-11.7:/include -isystem /home/strh/anaconda3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu.c -o prroi_pooling_gpu.o
FAILED: prroi_pooling_gpu.o
c++ -MMD -MF prroi_pooling_gpu.o.d -DTORCH_EXTENSION_NAME=_prroi_pooling -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/TH -isystem /home/strh/anaconda3/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda-11.7:/include -isystem /home/strh/anaconda3/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu.c -o prroi_pooling_gpu.o
In file included from /media/strh/MyDrive/Track/OtherTracking/pytracking/ltr/external/PreciseRoIPooling/pytorch/prroi_pool/src/prroi_pooling_gpu.c:15:
/home/strh/anaconda3/lib/python3.10/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
5 | #include <cuda_runtime_api.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/home/strh/anaconda3/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/run_video.py", line 38, in <module>
main()
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/run_video.py", line 34, in main
run_video(args.tracker_name, args.tracker_param,args.videofile, args.optional_box, args.debug, args.save_results)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/run_video.py", line 20, in run_video
tracker.run_video_generic(videofilepath=videofile, optional_box=optional_box, debug=debug, save_results=save_results)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/evaluation/tracker.py", line 395, in run_video_generic
out = tracker.track(frame, info)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/evaluation/multi_object_wrapper.py", line 165, in track
out = self.trackers[obj_id].initialize(image, init_info_split[obj_id])
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/tracker/dimp/dimp.py", line 84, in initialize
self.init_classifier(init_backbone_feat)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../pytracking/tracker/dimp/dimp.py", line 573, in init_classifier
self.target_filter, _, losses = self.net.classifier.get_filter(x, target_boxes, num_iter=num_iter,
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/models/target_classifier/linear_filter.py", line 94, in get_filter
weights = self.filter_initializer(feat, bb)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/models/target_classifier/initializer.py", line 164, in forward
weights = self.filter_pool(feat, bb)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/models/target_classifier/initializer.py", line 45, in forward
return self.prroi_pool(feat, roi1)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/external/PreciseRoIPooling/pytorch/prroi_pool/prroi_pool.py", line 28, in forward
return prroi_pool2d(features, rois, self.pooled_height, self.pooled_width, self.spatial_scale)
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/external/PreciseRoIPooling/pytorch/prroi_pool/functional.py", line 44, in forward
_prroi_pooling = _import_prroi_pooling()
File "/media/strh/MyDrive/Track/OtherTracking/pytracking/pytracking/../ltr/external/PreciseRoIPooling/pytorch/prroi_pool/functional.py", line 30, in _import_prroi_pooling
_prroi_pooling = load_extension(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/strh/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension '_prroi_pooling'
Edit: I've changed the CUDA path in .bashrc but I have the same error.
setarekhosravi commented
I think the problem is PyTorch and CUDA versions. I should install the exact version that was used in this project.
eliafranc commented
Depending on which CUDA version you need, you might be able to use the docker image I wrote that works with pytracking. I used version 12.2 but if you need a lower one you can try to change it in the Dockerfile from FROM nvidia/cuda:12.2.0-devel-ubuntu20.04 to FROM nvidia/cuda:11.7.1-devel-ubuntu20.04.
setarekhosravi commented
@eliafranc Thank you, I'll try it.
setarekhosravi commented
I found that the problem is exactly the PyTorch version. By downgrading Pytorch from 2.0.1 to 1.13.0, the problem has been solved.
@eliafranc I used your tips and now I can use the repo, Thank you very much for your patience and attention.
ThePengg commented
No module named 'prroi_pool'
Obadahheyari commented
same problem please help me
eliafranc commented
did you initialize the submodule? @ThePengg @Obadahheyari
run git submodule update --init --recursive