m-tassano/fastdvdnet

Training takes too long

Closed this issue · 1 comments

SiddharthRaoA commented

Hi, I tried training your model on a Nvidia GTX 1050 GPU, but it takes around 5-6 days to complete the training (which seemed too long for the size of the dataset and model). It was trained on the DAVIS-train-dataset provided as a link in the README. Hence I wanted to know the estimated training time and the hardware on which it was trained.

Also, I used DALI is 0.10.0 as mentioned in the README. Could a newer version of DALI improve the speed in any way?

Thanks for your help!

m-tassano commented

Hi, it takes me 4-5 days to train a model on a Titan Xp. You can try augmenting the batch size if it fits in your GPU memory.

As for the newer version of DALI, it could improve the runtimes, but I don't know for sure if fairness, I haven't tried.

Hope this helps!