AKASH2907/deepfakes_video_classification

Getting loss as nan in train_triplets_semi_hard.py

yksolanki9 opened this issue · 4 comments

yksolanki9 commented

On training the network, I am getting loss as nan and a valid value for validation loss which decreases with epochs. Any idea as to why I am getting loss as nan and how to avoid it?

AKASH2907 commented

Try increasing your batch size. What's your current batch size?

yksolanki9 commented

I increased it to 64 and 128. Still getting nan loss.

AKASH2907 commented

Try 256 and 512 too. It happened with me also but when I increased the batch size nan loss disappeared.

yksolanki9 commented

That worked. Thanks!