关于训练第11个epoch

[wenhaixi]

用非常非常小的数据集想测试下，选用了20epoch，从10个后开始微调backbone，训练时候，前十个没问题，可以正常，但是到第11个报错如下：
Traceback (most recent call last):
File "C:/Users/Administrator/Desktop/python/siamban-master/tools/train.py", line 312, in
main()
File "C:/Users/Administrator/Desktop/python/siamban-master/tools/train.py", line 307, in main
train(train_loader, dist_model, optimizer, lr_scheduler, tb_writer)
File "C:/Users/Administrator/Desktop/python/siamban-master/tools/train.py", line 203, in train
outputs = model(data) # 此处进行前向传播进行计算.
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\utils\distributed.py", line 43, in forward
return self.module(*args, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\models\model_builder.py", line 76, in forward
xf = self.backbone(search)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\models\backbone\resnet_atrous.py", line 192, in forward
p3 = self.layer3(p2)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
input = module(input)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\models\backbone\resnet_atrous.py", line 104, in forward
residual = self.downsample(x)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
input = module(input)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
return self._conv_forward(input, self.weight)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\conv.py", line 420, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

请问这是什么问题呢？

[zeduchen]

Fine-tuning the backbone requires a large amount of video memory. You can check if there are other processes occupying the video memory.