yzd-v/cls_KD

Some question about ViTKD

peiyingxin opened this issue · 1 comments

peiyingxin commented

Hi, thanks for sharing your great work!
I have some question about your work:

  1. where are you get your deit3-base model? Offical model is 85.7 top1 accuracy in ImageNet-1K, paper deit3-base model is 85.48, in addition, official model state_dict is not same with your defined deit3 model state_dict, so you had modified it?
  2. I had used vit-base model from mmcls with 85.43 top1 accuracy distill deit-small from scratch. only got 80.04 top1 accuracy, witch below baseline 80.69, deit3-base model struct is same with vit-base, I'm confused why got this result?

hoping for your reply
thank you.

yzd-v commented

1.The weight is transferred from official weight. Beacuse of the environment, the accuracy maybe a little bit different.
2. ViT-Base is trained with 384x384. While DeiT-S is trained with 224x224. Please be more careful.