Question about reproduce result
Closed this issue · 2 comments
Hi!
I reproduce the TAP(w/o others) and the final accuracy is about 46.2% on the validation set. But it is reported the 49.91% on val set in the paper. Are there any details that I ignored? Or what is the reason for that?
Of course, due to insufficient memory, I can only set the batch size to 32, which is different from 128 in the paper.
Thanks a lot!
Hi @JayZhu0104 ,
Could you please provide additional details? The first step in debugging could be to separate the pre-training and fine-tuning steps. E.g., 1. whether fine-tuning from the released pre-training work? 2. what is the training from scratch baseline on your side (the 44.50 in the table)?
The smaller batch size could influence the performance. You could check its influence by comparing the training from scratch baseline (44.50) in the table.
Feel free to send me an email if I wasn't checking the git repo that frequently :)
Best,
Zhengyuan
@JayZhu0104 May I know how many GPUs you're using for training and the corresponding training time? Thanks!