how many hours to train 1000 epochs

Question

how many hours to train 1000 epochs

AugustusMJB opened this issue 5 years ago · 9 comments

hi,
with batchsize 128 and gtsrb's 50k images, how many hours do you used to train the 1000 epochs.
doese 1000 epochs necessary?

Answer 1 · 2019-07-10T05:02:47.000Z

Hi, Thank you for your interest. 1000 epoch is not necessary. The final model used in the paper is determined by a performance of the validation set (different from the test set). I ran around 1000 training epochs to make sure the validation performance saturates. The best model for the test set during 1000 epochs (by evaluating the performance of the test set every epoch) should be better than the model selected from the validation set, but that is the test set hacking. (Using the test set for validation during the training is not fair.) So the model is selected by the best validation performance. 1000 epochs is used just to ensure the validation performance saturation.

Answer 2 · 2019-07-10T05:06:57.000Z

I missed some of your questions. For gtsrb->tt100k I didn't use a separate validation set. The dataset was not large enough to split. I just reported best performance during the training stage. To run 1000 epochs, i do not remember the exact time, but it takes less than a day. Approximately around 12~16 hours i guess.

Answer 3 · 2019-07-10T05:17:20.000Z

The image size is small. So memory usage is not much too. I used one 1080ti to train and batch 128 takes only around 3~4 GB. I believe you can use larger batch-size e.g. 256. I just used 128 because the 128 number is the one of common batch size in the papers. Another reason I used batch-size 128 even there is enough memory space in GPU is because baseline methods take more memory than my approach. I used matching networks as a comparison baseline. Matching networks requires a lot of GPU memory compared to my approach. So I cannot use large batch-size. In order to compare fairly, I just used the same batch-size of 128 with matching networks.

Answer 4 · 2019-07-10T05:34:42.000Z

is the default setting ok for one gpu to train batchsize 128?
I use 8 gpus to train the batchsize 128, and 1st gpu has used 15g memory.
I am not sure what wrong with my reproduction for gtsrb2TT100k

Answer 5 · 2019-07-10T05:49:29.000Z

Try to run the code with one GPU. If memory problem happens, then let me know. I will check the git code again.

Answer 6 · 2019-07-10T05:56:03.000Z

hi, I have tried to use 1 gpu for training, it told me out of memory

Answer 7 · 2019-07-10T05:57:01.000Z

I will check the code tonight. Thanks for pointing out.

Answer 8 · 2019-07-11T14:36:40.000Z

Hi, I could not reproduce the memory error in my server.
What PyTorch version are you using?
The problem can be due to a different PyTorch version.

Answer 9 · 2019-07-12T06:36:02.000Z

Hello，the "out of memory" problem is solved, I am not very clear about the reason, maybe it's a server problem. But the training speed is still slow, I have not found the reason yet. It takes 1.5 hour to train 8 epochs, which means I need 15 days for training 2000 epochs.
I will check it later, I will contact you if in need,
thank you very much