traveller59/spconv

cuda execution failed with error 8: invalid device function

Closed this issue · 6 comments

fangjin-cool commented

When I run the test code , I get an error, which shows "cuda execution failed with error 8", and I print the error code ,it means "invalid device function".
Also , this error occurs when I run the code of second, Can anybody help me?
Cuda 9.0 + Gcc 5.5

cvnerds commented

I have run into this error under the following circumstances:

  • Circumstance: non_max_suppression is missing from spconv_utils
  • Therefore from spconv_utils import non_max_suppression fails
  • I copied and installed a working wheel from a local machine to the remote machine (spconv-1.1-cp37-cp37m-linux_x86_64.whl)
  • Now the from spconv_utils import non_max_suppression does not fail anymore
  • However, I also get error 8 during inference in indice.cu
  • => Something seems fishy with some CUDA-related functions
    pred = self.net(example)[0]
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/second.pytorch/second/pytorch/models/voxelnet.py", line 363, in forward
    preds_dict = self.network_forward(voxels, num_points, coors, batch_size_dev)
  File "/workspace/second.pytorch/second/pytorch/models/voxelnet.py", line 332, in network_forward
    voxel_features, coors, batch_size)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/second.pytorch/second/pytorch/models/middle.py", line 475, in forward
    ret = self.middle_conv(ret)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/spconv/modules.py", line 130, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/spconv/conv.py", line 177, in forward
    use_hash=self.use_hash)
  File "/opt/conda/lib/python3.7/site-packages/spconv/ops.py", line 93, in get_indice_pairs
    stride, padding, dilation, out_padding, int(subm), int(transpose), int(use_hash))
RuntimeError: spconv/src/spconv/indice.cu 60
cuda execution failed with error 8
turboxin commented

I ran into the same error of "cuda execution failed with error 8: invalid device function", after I re-compiled the spconv wheel file and re-installed it, it didn't fail again.

  • 👍1
Deleted user commented

I too got stuck with thesame problem. Can you kindly tell us how to solve this problem.. Kindly do the needful

chowkamlee81 commented

@jingleFun
@turboxin

Iam still facing this issue. Kindly let me know if you have solved it.
My configurations are below:

I had ubuntu18.0.4, cuda10.2 , torch=1.20 torch vision=0.4.0,

Yingfan99327 commented

I encounted such a problem too, I recompiled the wheel file and reinstalled it, but it didn't work, too. Have any of you got a good solution?