Reproduce validation set evaluation results with provided checkpoint

[eladb3]

Hi,
I'm trying to reproduce validation set evaluation results using the provided checkpoint, I get lower results than the results reported in your paper:

TEST: [INFO: logging.py: 67]: json_stats: {"noun_top1_acc": "24.40", "noun_top5_acc": "45.02", "split": "test_final", "verb_top1_acc": "41.44", "verb_top5_acc": "75.69"}

I used EPIC_100_validation.pkl file as the test split file, and set TEST.CHECKPOINT_FILE_PATH to SlowFast.pyth (the weights file provided).
Should I be able to reproduce the reported results that way?

Thanks

[ekazakos]

Hello,

Thanks for using the code! To double check I just re-run the test code using the same checkpoint (SlowFast.pyth) and EPIC_100_validation.pkl , and I get same results as the paper:

{"noun_top1_acc": "50.02", "noun_top5_acc": "75.62", "split": "test_final", "verb_top1_acc": "65.56", "verb_top5_acc": "90.00"}

So the answer is yes, you should be able to re-produce the results using these. Did you also use --cfg configs/EPIC-KITCHENS/SLOWFAST_8x8_R50.yamlas input argument for testing?

[eladb3]

Thank you very much for the fast answer

[iranroman]

Hello, I too have been able to reproduce the baseline results on noun and verb. Thank you very much for making this model, checkpoints and code available.

However, is there a flag that I can raise to also get the accuracy at the action level (i.e. whether verb+noun are both correct)?

[ekazakos]

Hi,

There is no such flag in the testing code. But you can use multitask_topk_accuraccies (link) for this purpose. For an example of how to use it, have a look here.

[iranroman]

Thank you very much. I modified the code to include that and it worked! I was able to replicate all the results (verb, noun, and action).