Turn cudnn.deterministic to False lead to faster and better trainning

Question

Turn cudnn.deterministic to False lead to faster and better trainning

Closed this issue 3 years ago · 4 comments

Hi, thank for this great work. What I have found is that turn cudnn.deterministic to False in line 49 of file main.py make the training way more faster and sometimes leads to better accuracy of the final model. I experience this phenomena with COCO, VOC and even my private dataset. Do you think this is a bug or something ?

Answer 1 · 2021-08-04T13:27:01.000Z

No, the "cudnn.deterministic" switch is used to repeat the result of your experiments. You can search for more information about this.

Answer 2 · 2021-08-06T03:08:20.000Z

So you only use cudnn.deterministic only for experiments and inference things and turn it off when training? Is that correct ?

Answer 3 · 2021-08-09T13:03:33.000Z

No, "make the experiment reproducible" means you can train the same model when you turn on this switch in the TRAINING stage.

Answer 4 · 2021-08-10T13:51:38.000Z

Ah OK that's make sense. Thank you. So I'm close this issue now