yoshitomo-matsubara/torchdistill

If the Teacher model is different from Student model, how can I use this framework？

topbookcc opened this issue · 2 comments

topbookcc commented

Hi. Thanks a lot for the great framework. I want to know if the student model is different from the Teacher model. For example, the Teacher model is BERT and the Student model is RNN(small model) .How should I use this framework?
Could you please tell me how I can implement that? Thanks

yoshitomo-matsubara commented

Hi @topbookcc ,

Yes, you could use this framework for such a configuration.
Assuming the student model (RNN) is an instance of nn.Module (implemented with PyTorch), can you describe the interface of RNN? Specifically, what would you like to feed to the student model (and for which task)?

If it is just a list of tokens (produced by the teacher's tokenizer) for GLUE tasks (i.e., output = your_rnn_model(*tokens)), I guess it should be easy.

yoshitomo-matsubara commented

@topbookcc Closing this issue as I don't see any follow-up for a while.
Open a new Discussion (not Issue) if you still have questions