How to duplicate the performance reported in your paper?

Hello, thank u for sharing your code.
I want to duplicate your model performances reported in your paper.
(Table 1- 9nd row: SMNA + FAST (beam) , 6nd row : SMNA + FAST (Greedy))

Would you please tell me how to duplicate the performance?

And I followed your readme(6th command).
It made two files(cache_val_unseen40False.json, search_val_unseen40False.json).
And I ran "Training Reranker.ipynb" file ,but i encountered some problem.
"FileNotFoundError: [Errno 2] No such file or directory: 'cache_train40False.json'"
There is a line about this in .ipynb
[train_data, val_seen, val_unseen] = load_data(['cache_train40False.json','cache_val_seen40False.json','cache_val_unseen40False.json'])
I think this code needs additional files(train40False.json file, val_seen40False.json)
How to get these file?
Where i should modify? ==> I fix this problem by modify the run_search.py.
Now, I got cache_train40False.json, cache_val_seen40False.json, cache_val_unseen40False.json, search_train40False.json, search_val_seen40False.json, search_val_unseen40False.json

But, the another problem still occurs.
I got issue on (line #40) Training Reranker.ipynb(below)
_, world_states, actions, sum_logits, mean_logits, sum_logp, mean_logp, pm, speaker, scorer = candidate

ValueError: not enough values to unpack (expected 10, got 4)
I printed the candidate(value) but, It only includes "traj"

How do i fix this problem?
Please answer to me.
Thank you!

Hi, thanks for your interest!
To reproduce the result:
a) For FAST - Greedy, you would need a trained seq2seq model and you DON't need a goal reranker.
b) For FAST - BEAM, you would need a trained seq2seq model and you will need to train a reranker.

The README, step 5, last command guides you to reproduce a):

python tasks/R2R/run_search.py --job search
                               --load_follower tasks/R2R/experiments/smna/snapshots/[name of the latest model]
                               --max_episode_len 40
                               --K 20
                               --logit 
                               --experiment_name FAST_short
                               --early_stop

Thank you for your reply.
I did step 5 already but, it occurs an error at step 6, Training Reranker.ipynb (above).
How do i get 10 values?

Hey, nice to see you are online! I am double checking my code right now and want to verify the steps before I get back to you. But, meanwhile, maybe you can give me an example of your cache json file? (less xxxx.json?)

Thank you for kindly reply.
I got 6 files after i did the command (step 6)
[ cache_train40False.json, cache_val_seen40False.json, cache_val_unseen40False.json, search_train40False.json, search_val_seen40False.json, search_val_unseen40False.json ]
For example, in case of "cache_val_seen40False.json " it contains like these info
{"2899_0": {"traj": {"instr_id": "2899_0", "instr_encoding": [27, 363, 10, 6, 10, 3, 58, 12, 3, 33, 5, 59, 24, 8, 13, 14, 83, 27, 85, 10, 3, 58, 12, 3, 195, 172, 33, 4, 83, 27, 85, 10, 3, 58, 12, 3, 33, 59, 24, 9, 7, 5, 22, 15, 3, 687, 15, 3, 130, 96, 3, 33, 5, 401, 402, 33, 4], "trajectory": [ ["803fe63ef23e4c06809c1722bb3e198e", 3.141592653589793, 0.0], ["1d789dc0ab3a48c8aa051f006d8ae874", 4.1887902047863905, 0.0], ..... "cand": [["2899_0", [["ZMojNkEp431", .....
After i saw your code( follower.py), I think it is related with

FAST/follower.py

Line 935 in 87eac10

if hasattr(self, 'cache_candidates'):

Ok I think we are onto something. That is not the expected file format for cache_XXXX.json. I am trying to upload everything I had for reproducing these results to Google Drive. Meanwhile, let me check the code to cache files ...

Thank you so much.
I'll check again too.

Ah, I made a mistake.
The 6 files were after i did the command (step 6, first command)
python tasks/R2R/run_search.py --job cache --load_follower tasks/R2R/experiments/smna/snapshots/[name of the latest model] --max_episode_len 40 --K 20 --logit --experiment_name cacheFAST

Hi, I just made two changes:

A small commit that should fix your error in generating & using the intermediate files. Commit
I uploaded my trained model and my intermediate files to the Google Drive. These should guarantee that you can generate the same numbers as in the paper.

Let me know if you still have any problem?

Thank you so much.
I'll check the commit.
If i have any problem, i'll send you an e-mail.