Getting Nan loss on MJSynth Dataset

Question

Getting Nan loss on MJSynth Dataset

YunusAbdi opened this issue 4 years ago · 4 comments

I downloaded the MJSynth dataset and followed the instructions but while training at the first epoch the loss suddenly changes to Nan, i tried adding regularization and norm clipping but both didn't fix the problem

Also the sequence accuracy is 0.0 most of the time

Answer 1 · 2021-03-16T03:26:32.000Z

I downloaded the MJSynth dataset and followed the instructions but while training at the first epoch the loss suddenly changes to Nan, i tried adding regularization and norm clipping but both didn't fix the problem

Also the sequence accuracy is 0.0 most of the time

Hello, I met the same problem. Did you solve it?

Answer 2 · 2021-03-16T04:10:49.000Z

Please provide more information.
By the way, here is a log trained by the MJSynth dataset.

Answer 3 · 2021-04-12T08:23:47.000Z

I met the same problem. The sequence_accuracy is always 0.0, but the loss looks like going down normally.

Answer 4 · 2021-04-13T00:56:08.000Z

Don't worry. it's normal. please continue your training.

…

在 2021年4月12日，下午4:24，ddddddreamcastle ***@***.***> 写道： I met the same problem. The sequence_accuracy is always 0.0, but the loss looks like going down normally. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.