Confusion about training time?

[rightchose]

How long it takes to train the model one epoch?

[JUGGHM]

Thanks for your interest! It takes about 4 hours/epoch in the first stage and 2 hours/epoch in the third stage in our experimental environments.

[rightchose]

Thanks for your answer! But if the settting of epoch in first stage is set to 30(in paper) or 100(in code default), it may be too long. So in pratice, did you have some ways to reduce the time or quickly to get feedback?

[JUGGHM]

Thanks for your answer! But if the setting of epoch in first stage is set to 30(in paper) or 100(in code default), it may be too long. So in practice, did you have some ways to reduce the time or quickly to get feedback?

It is surely time consuming for training the full model with only 2x11G GPUs. For faster training, you could try larger batch size and tuning the learning rate hyper-parameters if more computational resources are available. In addition, strategies such as training smaller models (reduce the channel numbers to half) or training on smaller dataset (use a subset of the full dataset) are recommended.

[rightchose]

Thanks a lot! I will try it.

[rightchose]

I'm now training the model in the stage 3, so I want to know in stage3 how many epoch you get the model convergent?

[JUGGHM]

You could refer to this question. If you are equipped with more computational resources, larger resolution cropping and larger batch sizes are strongly recommended to accelerate training. And the learning rate could be adjusted accordingly.

[rightchose]

Thanks！ I will try this solutions.