How to reproduce results from the paper?
johnsyin opened this issue · 4 comments
Is it just the testing result in the last epoch using default parameters? I have tried to run add_test.py and below is the result i get for the 10 epochs.
Test set: Average loss: 0.168699
Test set: Average loss: 0.001142
Test set: Average loss: 0.000922
Test set: Average loss: 0.000345
Test set: Average loss: 0.000143
Test set: Average loss: 0.000188
Test set: Average loss: 0.000121
Test set: Average loss: 0.000028
Test set: Average loss: 0.000244
Test set: Average loss: 0.000042
Which one should I use for benchmarking? In the paper, the result of TCN was 5.8e-5 but it seems like we can use 2.8e-5 or 4.2e-5 here.
Hi @johnsyin ,
Thanks for your interest in our work!
In our paper, I took the average of the test set results from the last 3 epochs. In your case, I think you can also do the same. An alternative way is to use annealing learning rate; after the learning rate is annealed to the bottom, the test result will also converge (whose result you can then report).
But given the numbers you provided, I think you have already reproduced the results :-)
Hi Jerry,
Loved your paper, very appreciate you open sourced the work! I am a MS student researching this method.
- The last 3 numbers I provided has a mean of 1e-4, why is it considered as result reproduced (5.8e-5)?
- If I can have your advice, do you think multivariate TCN with Squeeze-and-excitation weighted temporal block will work?
Regarding your questions:
-
It's mainly a problem of training instability, 2.8e-5 and 4.2e-5 are already below what I reported. Note that these are MSE errors and random guess gives you only ~0.17.
-
Not sure what exactly your design is, but multivariate TCN has already been widely applied and I think it'd be (generally) a good direction to pursue.
Thank you for your reply!