training loss become nan
Opened this issue · 2 comments
Hi, thanks for your amazing work!but I encountered some problems while reproducing the work .The training loss became nan during training process. If there are possible solutions to my problem? Thanks in advance!
37%|███▋ | 148/400 [2:54:15<4:53:23, 69.85s/it]train loss: 0.3699 embedding norm: 23.580 Triplets (all/active): 95.9/23.4 Mean dist (pos/neg): 0.000/0.000
37%|███▋ | 149/400 [2:55:25<4:52:58, 70.03s/it]train loss: nan embedding norm: nan Triplets (all/active): 95.7/17.5 Mean dist (pos/neg): 0.000/0.000
Sorry, I'm little busy these days.
Have you solved the problem?
or can you post the config.txt and the training code?
Thanks for your replaying!I use the same training code and config files with this project,i am wondering whether it is caused by my cuda version 11.1