Training so slow

Question

Training so slow

hoyden opened this issue 4 years ago · 3 comments

I trained the model with my own data, but it so slow! It took me nearly 5 days to train 30k steps. BTW, I used 4 Tesla P40. Do you know how to solve it?

Answer 1 · 2021-02-02T08:17:59.000Z

@hoyden what RTF do you have at Tesla P40?

Answer 2 · 2021-02-03T15:35:44.000Z

@ivanvovk RTF 0.618 from the training log

Answer 3 · 2021-02-03T15:42:30.000Z

@hoyden I guess there might be some problems with distributed training. Maybe try to restart training on a single GPU, it won't differ too much.