Loss become NaN when training the model after several epochs
WZH-0x408 opened this issue · 0 comments
WZH-0x408 commented
@beginner-sketch
I tried to reproduce the result on the NYC dataset on my computer under default settings (python main.py -mode train -version 0 cuda_name) but loss became NaN after several training epochs. I have checked the dependencies (Python 3.8+PyTorch 1.12.1 + NumPy 1.17.4) and tried to fix the issue by setting a lower learning rate (1E-5), but the problem persists.