locuslab/TCN

How to reproduce results from the paper?

johnsyin opened this issue · 4 comments

johnsyin commented

Is it just the testing result in the last epoch using default parameters? I have tried to run add_test.py and below is the result i get for the 10 epochs.

Test set: Average loss: 0.168699
Test set: Average loss: 0.001142
Test set: Average loss: 0.000922
Test set: Average loss: 0.000345
Test set: Average loss: 0.000143
Test set: Average loss: 0.000188
Test set: Average loss: 0.000121
Test set: Average loss: 0.000028
Test set: Average loss: 0.000244
Test set: Average loss: 0.000042

Which one should I use for benchmarking? In the paper, the result of TCN was 5.8e-5 but it seems like we can use 2.8e-5 or 4.2e-5 here.

jerrybai1995 commented

Hi @johnsyin ,

Thanks for your interest in our work!

In our paper, I took the average of the test set results from the last 3 epochs. In your case, I think you can also do the same. An alternative way is to use annealing learning rate; after the learning rate is annealed to the bottom, the test result will also converge (whose result you can then report).

But given the numbers you provided, I think you have already reproduced the results :-)

johnsyin commented

Hi Jerry,

Loved your paper, very appreciate you open sourced the work! I am a MS student researching this method.

  1. The last 3 numbers I provided has a mean of 1e-4, why is it considered as result reproduced (5.8e-5)?
  2. If I can have your advice, do you think multivariate TCN with Squeeze-and-excitation weighted temporal block will work?
jerrybai1995 commented

Regarding your questions:

  1. It's mainly a problem of training instability, 2.8e-5 and 4.2e-5 are already below what I reported. Note that these are MSE errors and random guess gives you only ~0.17.

  2. Not sure what exactly your design is, but multivariate TCN has already been widely applied and I think it'd be (generally) a good direction to pursue.

  • 👍1
johnsyin commented

Thank you for your reply!