liuzywen/RGBTCC

训练过程中验证集上MSE和MAE无变化

Chenguoz opened this issue · 3 comments

Chenguoz commented

如下所示，按readme正常构建代码以及缩放数据集后，代码可以跑通，但是我观察到模型在Val上的结果没有变化，请问这是什么原因。

05-22 22:38:28 Epoch 0 Train, Loss: 247.92, GAME0: 20.62 MSE: 53.37, Cost 25.3 sec
05-22 22:38:39 Epoch 0 Val, MSE: 89.84 MAE: 65.43, Re: 1.0000,Cost 7.8 sec
05-22 22:38:39 save best mse 89.84 mae 65.43 model epoch 0
05-22 22:39:15 Epoch 0 test, MSE: 104.40 MAE: 74.10, Re: 1.0000,Cost 36.5 sec
05-22 22:39:16 -----Epoch 1/499-----
05-22 22:39:36 Epoch 1 Train, Loss: 218.65, GAME0: 16.19 MSE: 26.49, Cost 19.7 sec
05-22 22:39:46 Epoch 1 Val, MSE: 89.84 MAE: 65.43, Re: 1.0000,Cost 7.5 sec
05-22 22:39:46 -----Epoch 2/499-----
05-22 22:40:03 Epoch 2 Train, Loss: 230.78, GAME0: 17.04 MSE: 27.57, Cost 16.3 sec
05-22 22:40:15 Epoch 2 Val, MSE: 89.84 MAE: 65.43, Re: 1.0000,Cost 9.2 sec
05-22 22:40:15 -----Epoch 3/499-----
05-22 22:40:31 Epoch 3 Train, Loss: 219.46, GAME0: 16.44 MSE: 26.87, Cost 16.7 sec
05-22 22:40:42 Epoch 3 Val, MSE: 89.84 MAE: 65.43, Re: 1.0000,Cost 7.4 sec
ElderWu commented

你好，这个是因为全局计数Token没有初始化好，需要重新跑一下，如果验证指标正常下降就表示正常了。

Chenguoz commented

感谢回复~我尝试过跑多次、设置不同的随机种子等，然而奇怪的是验证指标均没有正常下降过

ElderWu commented