microsoft/TAP

Validation Accuracy different from paper

soonchangAI opened this issue · 0 comments

Hi, the validation accuracy I calculated for the fine-tuned models are different from the paper.
Command:

python -m torch.distributed.launch --nproc_per_node 2  tools/run.py --tasks vqa --datasets m4c_textvqa --model m4c_split \
--config $config \
--save_dir $folder \
--run_type val \
--resume_file $finetuned_model \
training_parameters.distributed True 

I observed changing the batch size results in different values.

Val accuracy for batch size = 32 Val acc for batch size = 128 In paper
TextVQA TAP (base) 49.87 49.53 49.91
TextVQA TAP (additional data) 54.31 54.13 54.71