An error when during backward when using torch.cuda.amp.autocast

[RongchangLi]

Hi, here is my envs:
torch==1.8
cuda==11.1
The error report is:

File "/data/Disk_A/jiaye/lrc_code/ddp_std/main.py", line 368, in train scaler.scale(loss).backward() File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/tensor.py", line 245, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/autograd/__init__.py", line 145, in backward Variable._execution_engine.run_backward( File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/autograd/function.py", line 89, in apply return self._forward_cls.backward(self, *args) # type: ignore File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/cuda/amp/autocast_mode.py", line 233, in decorate_bwd return bwd(*args, **kwargs) File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/depthwise_conv3d.py", line 46, in backward grad_input, grad_weight, grad_bias = DWCONV_CUDA.conv_depthwise3d_backward_cuda( RuntimeError: expected scalar type Half but found Float
When set amp is False, it works well.
How can I resolve this problem? Looking forward to your kind reply.

[PeiqinZhuang]

Hi, here is my envs: torch==1.8 cuda==11.1 The error report is:

File "/data/Disk_A/jiaye/lrc_code/ddp_std/main.py", line 368, in train scaler.scale(loss).backward() File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/tensor.py", line 245, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/autograd/__init__.py", line 145, in backward Variable._execution_engine.run_backward( File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/autograd/function.py", line 89, in apply return self._forward_cls.backward(self, *args) # type: ignore File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/torch/cuda/amp/autocast_mode.py", line 233, in decorate_bwd return bwd(*args, **kwargs) File "/data/Disk_A/jiaye/.conda/envs/torch19/lib/python3.9/site-packages/depthwise_conv3d.py", line 46, in backward grad_input, grad_weight, grad_bias = DWCONV_CUDA.conv_depthwise3d_backward_cuda( RuntimeError: expected scalar type Half but found Float When set amp is False, it works well. How can I resolve this problem? Looking forward to your kind reply.

Hi, have you solved this problem yet? Right now, the only way I can do is to turn off mixed precision to skip this error.

[gungui98]

@RongchangLi @PeiqinZhuang Sorry for late reply, please give me example code that causes the error

[PeiqinZhuang]

@RongchangLi @PeiqinZhuang Sorry for late reply, please give me example code that causes the error

Hi, I just send you an invitation from slack. Hopefully, it would be helpful for convenient communication.