Error AtomicAdd running python setup.py build develop
Opened this issue · 2 comments
samueleruffino99 commented
I am trying to build faster_rcnn with ```python setup.py build develop``. I am getting this error (every time atomicAdd is used):
error: no instance of overloaded function "atomicAdd" matches the argument list
argument types are: (double *, double)
detected during instantiation of "void RoIAlignBackwardFeature(int, const T *, int, T, int, int, int, int, int, int, T *, const T *) [with T=double]"
The final error log is:
Traceback (most recent call last):
File "setup.py", line 59, in <module>
setup(
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 765, in build_extensions
build_ext.build_extensions(self)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
_build_ext.build_extension(self, ext)
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
objects = self.compiler.compile(
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 586, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1487, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/itet-stor/sruffino/net_scratch/conda_envs/handobjdet/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1824, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension```
I had the same environment as you mentioned, one possible issue could be with CUDA driver.
Do you have any idea on how to fix it?
PS: it builds perfectly when I am not using GPU
samueleruffino99 commented
I figured out that, since my coda capability is 5.0 at maximum, atomAdd is not supported (it is from 6.0 on) and I had to define it by myself. See https://stackoverflow.com/questions/37566987/cuda-atomicadd-for-doubles-definition-error for more explanation. In particular, I had to add at the beginning of RoiAlign_cuda.cu and RoiPool_cuda.cu the following lines of code:
/ Define custom atomicAdd when cuda capability is less then 6.00
#if __CUDA_ARCH__ < 600
__device__ double myatomicAdd2(double* address, double val)
{
unsigned long long int* address_as_ull =
(unsigned long long int*)address;
unsigned long long int old = *address_as_ull, assumed;
do {
assumed = old;
old = atomicCAS(address_as_ull, assumed,
__double_as_longlong(val +
__longlong_as_double(assumed)));
// Note: uses integer comparison to avoid hang in case of NaN (since NaN != NaN)
} while (assumed != old);
return __longlong_as_double(old);
}
#endif
```.
Moreover, pay attention that you should define them with different names (e.g. 1 and 2 for example) or define it in another file as global function.
viprob-ai commented
i followed your suggestion but still getting below error :
any help ?
error: argument of type "float *" is incompatible with parameter of type "double *"
thanks in advance