Having trouble getting the correct scale of RMSE&MAE
Closed this issue · 8 comments
Hi,
It seems like I'm having a little bit of trouble getting the result of RMSE & MAE during evaluation with released pretrained models (regardless of evaluation sets). The iMAE & iRMSE are very close to reported values, however, RMSE & MAE are below 1. I wonder if I run the script wrong or these two evaluation metrics are not computed correctly?
I use all default settings and only change the data path in params.json.
Hi @yzhao520,
Would you please tell me your RMSE & MAE results? So that I can test again from my end.
Yeah, thanks for your reply! This is what I got after running python run_eval.py -exp=exp_msg_chn -chkpt=workspace/exp_msg_chn/final_model.pth.tar -set=selval
{'train': {85898}, 'val': {6852}, 'selval': {1000}, 'test': {1000}}
< Evaluate mode ! >
loading checkpoint from : workspace/exp_msg_chn/final_model.pth.tar
Checkpoint was loaded successfully!
Evaluating on [selval] set, Epoch [0] !
Evaluation results on [selval]:
[Loss]: 0.73930324
[MAE()]: 0.22482962
[RMSE()]: 0.81707797
[iMAE()]: 0.99338598
[iRMSE()]: 2.47487096
[Time]: 0.0068
[Time_av]: 0.0255
Hi @yzhao520，
There is no problem with the results, which are the same as the figures we reported in README.md.
You might mean that the results are different from those we reported in the paper.
This is because we used 64 channels and we trained the final model on training + evaluation sets in the paper for the final test.
Here we updated the model and only used 32 channels, and we train the model with only training set.
We find that the 32-channel network performs similarly to the 64-channel network on the test set, but has a much smaller number of parameters and a shorter running time.
Sorry for the confusion. My question is about why the result I get is different from what reported in README.md.
On the selected validation set, the reported RMSE and MAE in README.md are 817.08 and 224.83. However, what I get here after running the code is 0.81707797 and 0.22482962. Apparently, these two results are different. Could you explain what's wrong here? Did I run the code in a wrong way or there is another way to interpret this value?
Thanks
Oh, sorry for the question. Now, I see the relation between these two values. So I have to time 1000 to get what you reported in the README.md.
Yes @yzhao520, I guess you have found out that output values from the code are evaluated in meters, but in the README.md, we reported the values in millimeters. Our representation was confusing. I've just modified MAE`` and RMSEinutils/ErrorMeMetrics.py ```.
Sorry for asking this silly question and really appreciate your help!
@yzhao520, That's okay! Feel free to tell me whatever confuses you.