o_ddim: Loss too large
Closed this issue · 5 comments
DS-77 commented
Hello Authors of CoPaint,
I'm running your model on a custom dataset but, I'm having a little trouble with the o_ddim algorithm. My images are 256x256 and I'm using the imagenet pre-trained weights for inpainting tasks (./checkpoints/256x256_diffusion.pt and ./checkpoints/256x256_classifier.pt). For my configurations, I'm using all the settings found in the imagenet.yaml file in the configs directory in the project. When I attempt to run the model, it only gets to the 24 / 249 step before the Loss becomes too large. I have the output of the last step before this happens:
023-07-03-18:18:44-root-INFO: step: 226 lr_xt 0.00040579 23/249 [00:25<03:57, 1.05s/it]
2023-07-03-18:18:44-root-INFO: grad norm: 9656.987 8915.656 3710.590
2023-07-03-18:18:45-root-INFO: grad norm: 6539.739 6339.173 1607.194
2023-07-03-18:18:45-root-INFO: Loss Change: 126943.812 -> 97192.695
2023-07-03-18:18:45-root-INFO: Regularization Change: 0.000 -> 30.761
2023-07-03-18:18:45-root-INFO: Learning rate of xt decay: 0.02631 -> 0.02663.
Everything works fine with the ddim and the other algorithms, but not with o_ddim.
Any advice or assistance would be greatly appreciated! Thank you for an excellent model!
Best
ghzhang233 commented
Hi,
In intuition, with optimize_xt.use_adaptive_lr_xt=true, the loss should keep decreasing in each step. It turns out that the loss is still decreasing. I wonder what it would be until the final step.
Besides, the learning rate plays an important role in the optimization. If the final loss is still very large (and the performance is bad), I would suggest trying other values of learning rate.
Let me know if there is any other question.
Regards,
Guanhua
DS-77 commented
Hello Guanhua,
Thank you so much for your response!
Honestly, I don't think it ever reaches the last step, at least not in my case. I've provided a small sample of the output that continues to print to the screen indefinitely.
023-07-05-13:21:50-root-INFO: step: 225 lr_xt 0.00042598 | 24/249 [00:27<03:57, 1.06s/it]
2023-07-05-13:21:50-root-INFO: grad norm: nan nan nan
2023-07-05-13:21:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00034.
2023-07-05-13:21:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00027.
2023-07-05-13:21:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00022.
2023-07-05-13:21:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00017.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00014.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00011.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00009.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00007.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00006.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00005.
2023-07-05-13:21:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00004.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00003.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00002.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00002.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00001.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00001.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00001.
2023-07-05-13:21:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00001.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00001.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:21:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:06-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:07-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:07-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:07-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:07-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:07-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:07-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:08-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:09-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:10-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:11-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:12-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:13-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:14-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:15-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:16-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:17-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:18-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:18-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:18-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:18-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:18-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:18-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:19-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:20-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:21-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:22-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:23-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:24-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:25-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:26-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:27-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:28-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:29-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:30-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:31-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:31-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:31-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:31-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:31-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:31-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:32-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:33-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:34-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:35-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:36-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:37-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:38-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:39-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:40-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:41-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:41-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:41-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:41-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:41-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:41-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:42-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:43-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:44-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:45-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:46-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:47-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:48-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:49-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:50-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:51-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:52-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:53-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:54-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:55-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:56-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:57-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:58-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:22:59-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:00-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:01-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:02-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:03-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:04-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
2023-07-05-13:23:05-root-INFO: Loss too large (122896.555->nan)! Learning rate decreased to 0.00000.
So far I've tried following learning rates with corresponding step at which the loss became too large:
- 0.05 --> 20/249
- 0.02 --> 24/249
- 0.01 --> 44/249
- 0.005 --> 66/249
- 0.002 --> 61/249
- 0.0025 --> 54/249
- 0.001 --> 50/249
Thanks again for your help!
Best,
DS
ghzhang233 commented
Hello DS,
Are you able to reproduce our results following the instruction in README? If so, is there any modification to the code? It seems to me that it might be a problem with the learning rate or loss. Maybe trying it with an even smaller learning rate (like 1e-4 or 1e-5) would help. Let me know if it still does not work out.
Regards,
Guanhua
DS-77 commented
Hi Guanhua,
I think you might have found the problem! Initially I ran the examples on a colleague's computer, but did not re-run the code on my local computer before continuing with our project. I believe my GPU may be to blame. I've reached out to my university and they have resources I can use to run the code on a more sufficient GPU.
We haven't made any alterations to the code, but we have made changes to the configuration file for our dataset. If it is possible, is there a way I could add a counter or a way to exit the code after a certain amount of iterations in the p_sample_loop? I will run the code with all of the default configurations, but I want to make sure the code does not run indefinitely; I won't be able to see the output right away.
I also tried the smaller learning rate (1e-5) as well and it did significantly better. The code reached 127/249 steps before permanently staying at "Loss too large! Learning rate decreased to 0.00000".
I will re-run the code and I'll let you know if the better GPU help. Thank you so much for working with me and all of your help!
Best,
DS
DS-77 commented
My Hardware was the issue. Thank you so much for your research and your assistance!