在训练的时候出现了这个错误，Found Inf or NaN global norm，哪个大牛帮忙看一下怎么破？
guang1234 opened this issue · 1 comments
guang1234 commented
InvalidArgumentError (see above for traceback): Found Inf or NaN global norm. : Tensor had Inf values
[[node VerifyFinite/CheckNumerics (defined at C:\Users\xxx\seq2seq-couplet-master1\model.py:79) = CheckNumericsT=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
wb14123 commented
Try to restart the training from checkpoint