he-dhamo/simsg

Some problems in training

VicZlq opened this issue · 1 comments

Hello! Thank you very much for your wonderful project! I have solved the problem of the last route. However, I found the training was slow, and it took me about 15 days to run 200 epoch with a v100. What equipment do you use for project training? How long will it take to train? Is there a way to speed up training? Thank you!

Hello! We measured training iterations instead of epochs. All models on VG were trained for 300k iterations and on CLEVR for 40k iterations. Training on an Nvidia RTX 2080 Ti GPU, for images of size 64 × 64 takes about 3 days for Visual Genome and 4 hours for CLEVR.