salesforce/ALBEF

Some results issue

viyjy opened this issue · 4 comments

viyjy commented

Hi Junnan, I have the following questions about the result. Hope that you can help to clarify them, thanks.

  1. VQA:
    I get the result folder after fine-tuning on VQA dataset. Which json file should I use to get test-dev and test-std?
    image

  2. SNLI-VE:
    This is the log file after fine-tuning on SNLI-VE dataset. You didn't update the best-epoch, so it is always 0. Should I pick the row which has the best val accuracy as the final result?
    image

  3. Grouding
    This is the log file after fine-tuning on Ref-COCO. Should I pick the row which has the best val_d as the final result?
    image

  4. NLVR2
    This is the log file after fine-tuning on NLVR2, but I did't find dev and test-P as shown in your paper, any idea?
    image

LiJunnan1992 commented

Hi, thanks for your interest. Here are my answers.

  1. VQA: vqa_result_epoch7.json is the final result which collects results from all ranks.
  2. SNLI-VE: thanks for spotting my mistake. Yes you should use the row with best val_acc.
  3. Grounding: if I remembered correctly, in the paper I just reported the last epoch's result.
  4. NLVR: dev is val, test-P is test.
viyjy commented

Thanks very much!

haoshuai714 commented

Hello! Can NLVR2 datasets be shared?