A Distilation version of PhoBERT
huydang2k opened this issue · 1 comments
Hi, I am trying to get a distillation version of PhoBERT.
Is there any pre-trained of it or I have to train myself?
I found available code to train a distillation versions of transformers-models from tranformers package.
Link: https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation
I tried, but it doesn't seem to work well with PhoBert, due to some differences in tokenizer.
Any guide for me here. Thanks in advance.
You have to train a distilled version of PhoBERT.
Note that PhoBERT is based on RoBERTa, thus you would need to apply examples w.r.t. RoBERTa distillation. Also, please do word segmentation in the data (e.g. dump.txt) before binarizing it, and when binarizing the data, please select the corresponding PhoBERT tokenizer.