fgnt/pb_sed

RuntimeError: CUDA error

Closed this issue · 0 comments

Hi, how are you? I have a quetion.
I tried train_crnn.py, but CUDA error happened.
I attached error log file and, train_crnn.py code.

error_log.txt
train_crnn.py.txt

When I applied below modification, error not happen, but training time become long...
Do you know some workaround without below modification, or another way ?

--- modification ---
torch.backends.cudnn.enabled = False
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

--- error ---
Traceback (most recent calls WITHOUT Sacred internals):
File "C:\Users\XXXXXXX\Desktop\0_tec\210317_DCASE2020_Task4_3rd\pb_sed-master\pb_sed\pb_sed\experiments\dcase_2020_task_4\train_crnn.py", line 256, in train
trainer.train(train_iter, resume=resume)
File "c:\users\XXXXXXX\desktop\0_tec\210317_dcase2020_task4_3rd\pb_sed-master\padertorch\padertorch\train\trainer.py", line 435, in train
hook.close(self)
File "c:\users\XXXXXXX\desktop\0_tec\210317_dcase2020_task4_3rd\pb_sed-master\padertorch\padertorch\train\hooks.py", line 378, in close
self.finalize_summary(trainer)
File "c:\users\XXXXXXX\desktop\0_tec\210317_dcase2020_task4_3rd\pb_sed-master\padertorch\padertorch\train\hooks.py", line 301, in finalize_summary
self.summary = trainer.model.modify_summary(self.summary)
File "c:\users\XXXXXXX\desktop\0_tec\210317_dcase2020_task4_3rd\pb_sed-master\pb_sed\pb_sed\models\crnn.py", line 223, in modify_summary
image.flip(2), normalize=True, scale_each=False, nrow=1
RuntimeError: CUDA error: unspecified launch failure