-
This code is modified based on the repository developed by Hugging Face: Transformers v2.1.1
-
Prepare environment
pip install -r requirements.txt
-
Download the GLUE data by running this script and unpack it to directory GLUE_DIR
-
TASK_NAME can be one of CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE.
${GLUE_DIR}/${TASK_NAME}
-
The original Wiki data used in this project can be found here (It is called 'Raw text from English Wikipedia for general distillation')
-
The processed Wiki data can be generated by the method in TinyBERT using the following script from this repository
python pregenerate_training_data.py --train_corpus
${CORPUS_RAW} \ --bert_model $ {BERT_BASE_DIR}$
--reduce_memory --do_lower_case
--epochs_to_generate 3
--output_dir${CORPUS_JSON_DIR}$ ${BERT_BASE_DIR}$ includes the BERT-base teacher model, e.g., BERT-base-uncased
First train model on MNLI. Then finetune it on RTE. Finally evaluate it on RTE.
cd Code/run_yaml/
- $$AMLT_DATA_DIR/Data_GLUE/glue_data/{TASK_NAME}/ is data folder
- $$AMLT_DATA_DIR/Local_models/pretrained_BERTs/BERT_base_uncased/ contains the teacher and student initialization.
- Please create model_dir and download the pretrained BERT_base_uncased and put it here
Train_NoAssist_SeriesEpochs_NoHard_PreModel_RndSampl.yaml
- $$AMLT_DATA_DIR/Outputs/glue/MNLI/NoAssist/All_NoAug_NoHardLabel_PreModel/ contains the models trained on MNLI
- Epochs_{Epochs_TrainMNLI} is the different model trained on MNLI
- Please create the folder of $$AMLT_DATA_DIR/Outputs/glue/MNLI/NoAssist/All_NoAug_NoHardLabel_PreModel/ and put the output models of Step 1 here
Train_finalfinetuning_SpecificSubs_SeriesEpochs_NoAssist_NoHardLabel_PretrainedModel.yaml
- $$AMLT_DATA_DIR/Outputs/glue/{TASK_NAME}/NoAssist/All_FINETUNING_NoHardLabel_PreModel/SpecificSubs/ contains the models finetuned on RTE
- FinetuneEpochs_{Finetune_Epochs}EpochsMNLI{Epochs_TrainMNLI}Sub{Subs} is the different model finetuned on RTE
- Please create the folder of $$AMLT_DATA_DIR/Outputs/glue/{TASK_NAME}/NoAssist/All_FINETUNING_NoHardLabel_PreModel/SpecificSubs/ and put the output models of Step 2 here
Evaluate_SpecificSubs_NoAssist_NoHardLabel_PretrainedModel.yaml
First train model on Wiki. Then finetune it on RTE. Finally evaluate it on RTE.
cd Code/run_yaml/
- $$AMLT_DATA_DIR/English_Wiki/corpus_jsonfile_for_general_KD/ contains the processed Wiki data
Train_wiki_NoAssist_NoHard_PreModel_RndSampl.yaml