allenai/savn

reimplementation performance

xiaojun-chang opened this issue · 1 comments

xiaojun-chang commented

Hi,

Thanks for sharing the code. I strictly follow the instruction of this repo, but cannot get the performance reported. I can only get the following performance for SAVN.

{
"GreaterThan/1/spl": 0.13895125447326895,
"GreaterThan/1/success": 0.384,
"GreaterThan/5/spl": 0.11553156460140891,
"GreaterThan/5/success": 0.25898203592814373,
"done_count": 0.974,
"ep_length": 27.018,
"spl": 0.13895125447326895,
"success": 0.384,
"total_reward": 1.6536600000000015,
"total_time": 0.5675430691242218
}

Any suggestion?
Thanks.

Xiaojun

mitchellnw commented

To confirm you're using the same version of pytorch etc. ?

And you're running on the val set to choose the best model then running that on the test set?

If yes then I'm sorry -- I don't know why that would be. (If your intention is research) I would recommend reporting these numbers instead of ours, and just detail that you ran the code from the work.