Taking too much time in training

Question

Taking too much time in training

CS-savvy opened this issue 5 years ago · 3 comments

Model taking 2 hrs for one epoch having 2300 images and batch size is 1, but you guys have mentioned it took only 4 hrs to train page detection model which contains 1600 images for 30 epochs.

can someone tell me the reason?

Answer 1 · 2019-07-26T07:45:14.000Z

Can you give your GPU specs ?

Answer 2 · 2019-07-27T17:37:19.000Z

Thanks for replying @solivr
I am using Azure VM to train dhSegment model having NVIDIA-driver 390.116 GPU - Tesla K80 - 11 Gb gpu with 6 vcpu and 56 GB ram.

https://www.techpowerup.com/gpu-specs/tesla-k80.c2616

GPU -

CPU -

Answer 3 · 2021-07-24T18:50:59.000Z

@solivr @CS-savvy are there any updates here?? My instances are crashing due to the fact of memory even though I use 4 GPUs of 16 GB each. Can anyone suggest any improvement??