multi-GPU version
MichaelYu781 opened this issue · 4 comments
hi, @stepankonev !
Current code is with single gpu and the training speed is relatively slow (about 1.5 iteration per second from our side).
Do you have multi-GPU version?
Hi, @MichaelYu781 !
I was training the model on the single GPU so the multi-GPU version is not provided.
Best,
Got it. Thank you, sir!
Hi, @stepankonev !
What will the loss value finally converges to? In case the loss value is always positive, it will converges to zero. But when i train this model, the loss is negative. Although it gets down, but i don't know to which value it means the training model is close to completion. The attached photo is my training log, is this normal?
Thanks for your time!
Hello! Please, let's keep the discussion clean. This issue should be devoted to the multi-gpu version of the model. It would also be great if you share the config in the other issue, previously checking for duplicates. Thanks!
PS I believe now it should have converged to some point