linksense/LightNet

torch.FatalError: function_attributes(): after cudaFuncGetAttributes: invalid device function

HLH13297997663 opened this issue · 0 comments

The following error occurred when I ran the train_mobile_mvd:
/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/cuda/init.py:95: UserWarning:
Found GPU0 Tesla V100-SXM2-16GB which requires CUDA_VERSION >= 9000 for
optimal performance and fast startup time, but your PyTorch was compiled
with CUDA_VERSION 8000. Please install the correct PyTorch binary
using instructions from http://pytorch.org

warnings.warn(incorrect_binary_warn % (d, name, 9000, CUDA_VERSION))
^M 0%| | 0/9000 [00:00<?, ?it/s]['/workfs/lhhuang/segment/LightNet-master2/LightNet-master/scripts', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master/scripts', '/dat01/lhhuang/models/research', '/dat01/lhhuang/models/research/slim', '/dat01/lhhuang/models/research/delf', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python36.zip', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/lib-dynload', '/home/lhhuang/.local/lib/python3.6/site-packages', '/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master/datasets', '/workfs/lhhuang/segment/LightNet-master2/LightNet-master']

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

  1. Setting up DataLoader...

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

There are 66 labels in the config file
Found 18000 training images...
There are 66 labels in the config file
Found 2000 validation images...

+++++++++++++++++++++++++++++++++++++++++++++++++++++++

  1. Setting up Model...
  2. Model Training start...
    ^M 0%| | 1/9000 [00:04<12:23:13, 4.96s/it]^M> Epoch [1/256]: 0%| | 1/9000 [00:04<12:23:13, 4.96s/it]Traceback (most recent call last):
    File "train_mobile_mvd.py", line 348, in
    train(train_args, data_path, save_path)
    File "train_mobile_mvd.py", line 185, in train
    out_stg1, out_stg2, out_stg3 = model(images) # Here we have 3 output for 3 loss
    File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
    result = self.forward(*input, **kwargs)
    File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/models/mobilenetv2plus.py", line 169, in forward
    de_stg1 = self.aspp(stg8)[1]
    File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
    result = self.forward(*input, **kwargs)
    File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
    File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
    result = self.forward(*input, **kwargs)
    File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/misc.py", line 334, in forward
    x = self.in_norm(x)
    File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
    result = self.forward(*input, **kwargs)
    File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/bn.py", line 96, in forward
    self.activation, self.slope)
    File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/functions.py", line 97, in forward
    _act_forward(ctx, x)
    File "/workfs/lhhuang/segment/LightNet-master2/LightNet-master/modules/functions.py", line 41, in _act_forward
    _ext.leaky_relu_cuda(x, ctx.slope)
    File "/workfs/lhhuang/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/utils/ffi/init.py", line 180, in safe_call
    result = torch._C._safe_call(*args, **kwargs)
    torch.FatalError: function_attributes(): after cudaFuncGetAttributes: invalid device function