Loss NaN's
yousufzu opened this issue · 1 comments
yousufzu commented
Anyone else have this problem?
epoch 427: genloss 74090.617188 latloss 126.223068
epoch 428: genloss 77959.625000 latloss 83.587059
epoch 429: genloss 76822.828125 latloss 191.646500
epoch 430: genloss 166910.765625 latloss 58514.851562
epoch 431: genloss 131478.968750 latloss 59600.101562
epoch 432: genloss nan latloss nan
epoch 433: genloss nan latloss nan
epoch 434: genloss nan latloss nan
epoch 435: genloss nan latloss nan
epoch 436: genloss nan latloss nan
epoch 437: genloss nan latloss nan
The images decoded then go "dark" right after that.
yousufzu commented
Fixed this by replacing 1e-8 in the log()'s with 1e-6... but the AdamOptimizer's behavior is quite weird, the training goes well for some time and then the loss blows up for no reason. Looking at the results before the loss blows up, it clearly gets worse. Maybe this is just part of stochastic GD.
epoch 173: genloss 78835.515625 latloss 136.783737
epoch 174: genloss 79019.046875 latloss 128.061676
epoch 175: genloss 79939.609375 latloss 136.034576
epoch 176: genloss 79181.796875 latloss 161.920746
epoch 177: genloss 80438.468750 latloss 169.526703
epoch 178: genloss 78668.695312 latloss 163.433441
epoch 179: genloss 83566.210938 latloss 176.066071
epoch 180: genloss 80774.562500 latloss 149.725540
epoch 181: genloss 83706.460938 latloss 271.983826
epoch 182: genloss 79595.859375 latloss 176.206299
epoch 183: genloss 79057.421875 latloss 198.661102
epoch 184: genloss 79186.226562 latloss 215.053436
epoch 185: genloss 78844.296875 latloss 224.593140
epoch 186: genloss 78854.367188 latloss 193.455276
epoch 187: genloss 114669.484375 latloss 12226.135742
epoch 188: genloss 94797.234375 latloss 15731.478516
epoch 189: genloss 126349.937500 latloss 1008915.187500
epoch 190: genloss 126299.648438 latloss 1911658.250000
epoch 191: genloss 170011.468750 latloss 3560406528.000000
epoch 192: genloss 134594.984375 latloss 111272680.000000
epoch 193: genloss 135633.625000 latloss 53908908.000000
epoch 194: genloss 292594.531250 latloss 30987573248.000000
epoch 195: genloss 133974.078125 latloss 911580992.000000
epoch 196: genloss 132212.218750 latloss 5219494400.000000
epoch 197: genloss 133383.093750 latloss 2926796032.000000
epoch 198: genloss 122014.539062 latloss 2021904128.000000
epoch 199: genloss 119586.687500 latloss 166584528.000000
epoch 200: genloss 118031.164062 latloss 390215456.000000
epoch 201: genloss 114919.796875 latloss 388184000.000000
epoch 202: genloss 141407.953125 latloss 192784048.000000