whether logits self KD can be used alone

Question

whether logits self KD can be used alone

Closed this issue 2 years ago · 1 comments

Hi Wingcg!
Your idea has inspired me a lot. May I ask whether logits self KD can be used alone, instead of combining logits self KD and feature self KD, because the size of the model and the amount of calculation are taken into account.Thanks a lot!

Answer 1 · 2022-10-17T03:42:12.000Z

Thanks for your attention of our work. It is OK to apply logits self KD alone, but the performance may drop a lot.