RuntimeError: CUDA error: no kernel image is available for execution on the device
abhishek-peri opened this issue · 6 comments
abhishek-peri commented
Thanks for providing such a library!
I want to use the lietorch library in my project where I'm trying to optimize over the SE3 space. I've installed the required dependencies as stated. However, I'm receiving the below error when I try to run the ./run_test.sh script. So, I wanted to check if anyone can tell what could be the source of this error and how to rectify it?
Note: torch.cuda.get_device_name(0) returns correctly the GPU's name.
Testing lietorch forward pass (CPU) ...
- <class 'lietorch.groups.SO3'> Passed exp-log test
- <class 'lietorch.groups.SO3'> Passed inv test
- <class 'lietorch.groups.SO3'> Passed adj test
- <class 'lietorch.groups.SO3'> Passed act test
- <class 'lietorch.groups.RxSO3'> Passed exp-log test
- <class 'lietorch.groups.RxSO3'> Passed inv test
- <class 'lietorch.groups.RxSO3'> Passed adj test
- <class 'lietorch.groups.RxSO3'> Passed act test
- <class 'lietorch.groups.SE3'> Passed exp-log test
- <class 'lietorch.groups.SE3'> Passed inv test
- <class 'lietorch.groups.SE3'> Passed adj test
- <class 'lietorch.groups.SE3'> Passed act test
- <class 'lietorch.groups.Sim3'> Passed exp-log test
- <class 'lietorch.groups.Sim3'> Passed inv test
- <class 'lietorch.groups.Sim3'> Passed adj test
- <class 'lietorch.groups.Sim3'> Passed act test
Testing lietorch backward pass (CPU)...
- <class 'lietorch.groups.SO3'> Passed eye-grad test
- <class 'lietorch.groups.SO3'> Passed inv-grad test
- <class 'lietorch.groups.SO3'> Passed adj-grad test
- <class 'lietorch.groups.SO3'> Passed adjT-grad test
- <class 'lietorch.groups.SO3'> Passed act-grad test
- <class 'lietorch.groups.RxSO3'> Passed eye-grad test
- <class 'lietorch.groups.RxSO3'> Passed inv-grad test
- <class 'lietorch.groups.RxSO3'> Passed adj-grad test
-
<class 'lietorch.groups.RxSO3'> Passed adjT-grad test
- <class 'lietorch.groups.RxSO3'> Passed act-grad test
- <class 'lietorch.groups.SE3'> Passed eye-grad test
- <class 'lietorch.groups.SE3'> Passed inv-grad test
- <class 'lietorch.groups.SE3'> Passed adj-grad test
- <class 'lietorch.groups.SE3'> Passed adjT-grad test
- <class 'lietorch.groups.SE3'> Passed act-grad test
- <class 'lietorch.groups.Sim3'> Passed eye-grad test
- <class 'lietorch.groups.Sim3'> Passed inv-grad test
- <class 'lietorch.groups.Sim3'> Passed adj-grad test
- <class 'lietorch.groups.Sim3'> Passed adjT-grad test
- <class 'lietorch.groups.Sim3'> Passed act-grad test
Testing lietorch forward pass (GPU) ...
Traceback (most recent call last):
File "lietorch/run_tests.py", line 217, in <module>
test_exp_log(Group, device='cuda')
File "lietorch/run_tests.py", line 19, in test_exp_log
b = Group.exp(a).log()
File "/home/abhishek_peri/anaconda3/envs/nerfing/lib/python3.6/site-packages/lietorch-0.1-py3.6-linux-x86_64.egg/lietorch/groups.py", line 132, in log
return self.apply_op(Log, self.data)
File "/home/abhishek_peri/anaconda3/envs/nerfing/lib/python3.6/site-packages/lietorch-0.1-py3.6-linux-x86_64.egg/lietorch/groups.py", line 122, in apply_op
data = op.apply(cls.group_id, *inputs)
File "/home/abhishek_peri/anaconda3/envs/nerfing/lib/python3.6/site-packages/lietorch-0.1-py3.6-linux-x86_64.egg/lietorch/group_ops.py", line 12, in forward
out = cls.forward_op(ctx.group_id, *inputs)
RuntimeError: CUDA error: no kernel image is available for execution on the device
zachteed commented
What GPU are you using? Its possible I may need to update the compiler args in setup.py
abhishek-peri commented
Hi,
I'm using 'GeForce GTX TITAN X' as my GPU
zachteed commented
Ok it looks like you will need to use a slightly modified setup.py, but this should work
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
import os.path as osp
ROOT = osp.dirname(osp.abspath(__file__))
print(ROOT)
setup(
name='lietorch',
version='0.1',
description='Lie Groups for PyTorch',
author='teedrz',
packages=['lietorch'],
ext_modules=[
CUDAExtension('lietorch_backends',
include_dirs=[
osp.join(ROOT, 'lietorch/include'),
osp.join(ROOT, 'eigen')],
sources=[
'lietorch/src/lietorch.cpp',
'lietorch/src/lietorch_gpu.cu',
'lietorch/src/lietorch_cpu.cpp'],
extra_compile_args={
'cxx': ['-O2'],
'nvcc': ['-O2',
'-gencode=arch=compute_52,code=sm_52',
'-gencode=arch=compute_60,code=sm_60',
'-gencode=arch=compute_61,code=sm_61',
'-gencode=arch=compute_70,code=sm_70',
'-gencode=arch=compute_75,code=sm_75',
'-gencode=arch=compute_75,code=compute_75',
]
}),
CUDAExtension('lietorch_extras',
sources=[
'lietorch/extras/altcorr_kernel.cu',
'lietorch/extras/corr_index_kernel.cu',
'lietorch/extras/se3_builder.cu',
'lietorch/extras/se3_inplace_builder.cu',
'lietorch/extras/se3_solver.cu',
'lietorch/extras/extras.cpp',
],
extra_compile_args={
'cxx': ['-O2'],
'nvcc': ['-O2',
'-gencode=arch=compute_52,code=sm_52',
'-gencode=arch=compute_60,code=sm_60',
'-gencode=arch=compute_61,code=sm_61',
'-gencode=arch=compute_70,code=sm_70',
'-gencode=arch=compute_75,code=sm_75',
'-gencode=arch=compute_75,code=compute_75',
]
}),
],
cmdclass={ 'build_ext': BuildExtension }
)
abhishek-peri commented
Yes, this seems to solve my issue. Thanks for the quick response.
abhishek-peri commented
On a different note, I have seen in other issue that the backward operation for ToMatrix() is yet to be integrated. Is there any tentative timeline for its integration? Thanks.
zachteed commented
Just made an update so the .ToMatrix() function is now differentiable. I'm working on a differentiable FromMatrix function currently