cuda error

[dyl25984]

when I run it in docker container,it shows that:

Epoch: 1 [2020-09-02 08:56:59,475]
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
Traceback (most recent call last):
File "train.py", line 237, in
main()
File "train.py", line 226, in main
train(epoch)
File "train.py", line 143, in train
outputs = model(batch['image'])
File "/opt/conda/envs/centernet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/envs/centernet/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward
return self.module(*inputs[0], **kwargs[0])
File "/opt/conda/envs/centernet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/dyl/Centernet/nets/resdcn.py", line 220, in forward
x = self.conv1(x)
File "/opt/conda/envs/centernet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/envs/centernet/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:405