Pre-training has become an essential part for NLP tasks and has led to remarkable improvements. UER-py is a toolkit for pre-training on general-domain corpus and fine-tuning on downstream task. UER-py maintains model modularity and supports research extensibility. It facilitates the use of different pre-training models (e.g. BERT), and provides interfaces for users to further extend upon. UER-py also incorporates many mechanisms for better performance and efficiency. It has been tested on several Chinese datasets and should match or even outperform Google's TF implementation.
Update: Now ELMO (bilstm encoder + bilm target) is supported by UER. ELMO model pre-trained on mixed large corpus will be released soon. Word-based BERT is now available. Context-dependent word embedding (trained by BERT) is in particular suitable for polysemous words.
UER-py has the following features:
- Reproducibility. UER-py is able to reproduce the results of existing pre-training models (such as Google BERT).
- Multi-GPU. UER-py supports CPU mode, single GPU mode, and distributed training mode.
- Model modularity. UER-py is divided into multiple components: subencoder, encoder, target, and fine-tuning. Ample modules are implemented in each component. Clear and robust interface allows users to combine modules with as few restrictions as possible.
- Efficiency. UER-py refines its pre-processing, pre-training, and fine-tuning stages, which largely improves speed and needs less memory.
- SOTA results. Our works further improve the results upon Google BERT, providing new baselines for a range of datasets.
- Chinese model zoo. We are pre-training models with different corpora, encoders, and targets. Selecting proper pre-training models is beneficial to the performance of downstream tasks.
Python3.6 torch>=1.0 argparse
We use BERT model and Douban book review classification dataset to demonstrate how to use UER-py. We firstly pre-train model on book review corpus and then fine-tune it on classification dataset. There are three input files: book review corpus, book review dataset, and vocabulary. All files are encoded in UTF-8 and are included in this project.
The format of the corpus for BERT is as follows：
doc1-sent1
doc1-sent2
doc1-sent3
doc2-sent1
doc3-sent1
doc3-sent2
The book review corpus is obtained by book review dataset. We remove labels and split a review into two parts from the middle (See book_review_bert.txt in corpora folder).
The format of the classification dataset is as follows (label and instance are separated by \t):
label text_a
1 instance1
0 instance2
1 instance3
We use Google's Chinese vocabulary file, which contains 21128 Chinese characters. The format of the vocabulary is as follows:
word-1
word-2
...
word-n
First of all, we preprocess the book review corpus. We need to specify the model's target in pre-processing stage (--target):
python3 preprocess.py --corpus_path corpora/book_review_bert.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt \
--processes_num 8 --target bert
Pre-processing is time-consuming. Multi-process can largely accelerate the pre-processing speed (--processes_num). The raw text is converted to dataset.pt, which is the input of pretrain.py. Then we download Google's pre-trained Chinese model, and put it into models folder. We load Google's pre-trained model and train it on book review corpus. We should better explicitly specify model's encoder (--encoder) and target (--target). Suppose we have a machine with 8 GPUs.:
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --pretrained_model_path models/google_model.bin \
--output_model_path models/book_review_model.bin --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
--total_steps 20000 --save_checkpoint_steps 5000 --encoder bert --target bert
mv models/book_review_model.bin-20000 models/book_review_model.bin
Notice that the model trained by pretrain.py is attacted with the suffix which records the training step. We could remove the suffix for ease of use. Finally, we do classification. We can use google_model.bin:
python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
--epochs_num 3 --batch_size 32 --encoder bert
or use our book_review_model.bin, which is the output of pretrain.py：
python3 classifier.py --pretrained_model_path models/book_review_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
--epochs_num 3 --batch_size 32 --encoder bert
It turns out that the result of Google's model is 87.5; The result of book_review_model.bin is 88.1. It is also noticable that we don't need to specify the target in fine-tuning stage. Pre-training target is replaced with task-specific target.
BERT consists of next sentence prediction (NSP) target. However, NSP target is not suitable for sentence-level reviews since we have to split a review into two parts. UER-py facilitates the use of different targets. Using masked language modeling (MLM) as target could be a properer choice for pre-training of reviews:
python3 preprocess.py --corpus_path corpora/book_review.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt \
--processes_num 8 --target mlm
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --pretrained_model_path models/google_model.bin \
--output_model_path models/book_review_mlm_model.bin --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
--total_steps 20000 --save_checkpoint_steps 5000 --encoder bert --target mlm
mv models/book_review_mlm_model.bin-20000 models/book_review_mlm_model.bin
python3 classifier.py --pretrained_model_path models/book_review_mlm_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
--epochs_num 3 --batch_size 32 --encoder bert
It turns out that the result of book_review_mlm_model.bin is 88.3.
We could search proper pre-trained models in Chinese model zoo for further improvements. For example, we could download a model pre-trained on Amazon corpus (over 4 million reviews) with BERT encoder and classification (CLS) target. It achieves 88.5 accuracy on book review dataset.
BERT is really slow. It could be great if we can speed up the model and still achieve competitive performance. We select a 2-layers LSTM encoder to substitute 12-layers Transformer encoder. We could download a model pre-trained with LSTM encoder and language modeling (LM) + classification (CLS) targets:
python3 classifier.py --pretrained_model_path models/lstm_reviews_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
--epochs_num 3 --batch_size 64 --encoder lstm --pooling mean --config_path models/rnn_config.json --learning_rate 1e-3
We can achieve 86.5 accuracy on testset, which is also a competitive result. Using LSTM without pre-training can only achieve 80.2 accuracy. In practice, above model is around 10 times faster than BERT. One can see Chinese model zoo section for more detailed information about above pre-trained LSTM model.
Besides classification, UER-py also provides scripts for other downstream tasks. We could use tagger.py for sequence labeling:
python3 tagger.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/msra/train.tsv --dev_path datasets/msra/dev.tsv --test_path datasets/msra/test.tsv \
--epochs_num 5 --batch_size 16 --encoder bert
We could download a model pre-trained on RenMinRiBao (as known as People's Daily, a news corpus) and finetune on it:
python3 tagger.py --pretrained_model_path models/rmrb_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/msra/train.tsv --dev_path datasets/msra/dev.tsv --test_path datasets/msra/test.tsv \
--epochs_num 5 --batch_size 16 --encoder bert
It turns out that the result of Google's model is 92.6; The result of rmrb_model.bin is 94.4.
This project includes a range of Chinese datasets. Small-scale datasets can be downloaded at datasets_zh.zip. datasets_zh.zip contains 7 datasets: XNLI, LCQMC, MSRA-NER, ChnSentiCorp, and nlpcc-dbqa are obtained from Baidu ERNIE; Book review (from BNU) and Shopping are two sentiment analysis datasets. Large-scale datasets can be found in glyph's github project.
UER-py is organized as follows：
UER-py/
|--uer/
| |--encoders/: contains encoders such as RNN, CNN, Attention, CNN-RNN, BERT
| |--targets/: contains targets such as language modeling, masked language modeling, sentence prediction
| |--subencoders/: contains subencoders such as RNN, CNN, and different pooling strategies
| |--layers/: contains frequently-used NN layers, such as embedding layer, normalization layer
| |--models/: contains model.py, which combines subencoder, embedding, encoder, and target modules
| |--utils/: contains frequently-used utilities
| |--model_builder.py
| |--model_saver.py
| |--trainer.py
|
|--corpora/: contains corpora for pre-training
|--datasets/: contains downstream tasks
|--models/: contains pre-trained models, vocabularies, and config files
|--scripts/: contains some useful scripts for pre-training models
|
|--preprocess.py
|--pretrain.py
|--classifier.py
|--cloze.py
|--tagger.py
|--feature_extractor.py
|--README.md
The code is well-organized. Users can use and extend upon it with little efforts.
usage: preprocess.py [-h] --corpus_path CORPUS_PATH --vocab_path VOCAB_PATH
[--dataset_path DATASET_PATH]
[--tokenizer {bert,char,space}]
[--processes_num PROCESSES_NUM]
[--target {bert,lm,cls,mlm,nsp,s2s}]
[--docs_buffer_size DOCS_BUFFER_SIZE]
[--instances_buffer_size INSTANCES_BUFFER_SIZE]
[--seq_length SEQ_LENGTH] [--dup_factor DUP_FACTOR]
[--short_seq_prob SHORT_SEQ_PROB] [--seed SEED]
--docs_buffer_size and --instances_buffer_size could be used to control memory consumption in pre-processing and pre-training stages. --preprocesses_num n denotes that n processes are used for pre-processing. The example of pre-processing on a single machine is as follows：
python3 preprocess.py --corpus_path corpora/book_review_bert.txt --vocab_path models/google_vocab.txt \
--dataset_path dataset.pt --processes_num 8 --target bert
We need to specify the model's target in pre-processing stage since different targets require different data formats. Currently, UER-py consists of the following target modules:
- lm_target.py: language model
- mlm_target.py: masked language model (cloze test)
- nsp_target.py: next sentence prediction
- cls_target.py: classification
- s2s_target.py: supports autoencoder and machine translation
- bert_target.py: masked language model + next sentence prediction
If multiple machines are available, each machine contains a part of corpus. The command is identical with the single machine case.
usage: pretrain.py [-h] [--dataset_path DATASET_PATH] --vocab_path VOCAB_PATH
[--pretrained_model_path PRETRAINED_MODEL_PATH]
--output_model_path OUTPUT_MODEL_PATH
[--config_path CONFIG_PATH] [--total_steps TOTAL_STEPS]
[--save_checkpoint_steps SAVE_CHECKPOINT_STEPS]
[--report_steps REPORT_STEPS]
[--accumulation_steps ACCUMULATION_STEPS]
[--batch_size BATCH_SIZE]
[--emb_size EMB_SIZE] [--hidden_size HIDDEN_SIZE]
[--feedforward_size FEEDFORWARD_SIZE]
[--kernel_size KERNEL_SIZE] [--heads_num HEADS_NUM]
[--layers_num LAYERS_NUM] [--dropout DROPOUT] [--seed SEED]
[--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
[--bidirectional] [--target {bert,lm,cls,mlm,nsp,s2s}]
[--labels_num LABELS_NUM] [--learning_rate LEARNING_RATE]
[--warmup WARMUP] [--world_size WORLD_SIZE]
[--gpu_ranks GPU_RANKS [GPU_RANKS ...]]
[--master_ip MASTER_IP] [--backend {nccl,gloo}]
Notice that it is recommended to explicitly specify model's encoder and target. UER-py consists of the following encoder modules:
- rnn_encoder.py: contains (bi-)LSTM and (bi-)GRU
- cnn_encoder.py: contains CNN and gatedCNN
- attn_encoder.py: contains attentionNN
- gpt_encoder.py: contains GPT encoder
- bert_encoder.py: contains BERT encoder
- mixed_encoder.py: contains combinations of basic encoders, such as RCNN (RNN+CNN), CRNN (CNN+RNN)
The target should be coincident with the target in pre-processing stage. Users can try different combinations of encoders and targets by --encoder and --target.
There are two strategies for pre-training: 1）random initialization 2）loading a pre-trained model.
The example of pre-training on CPU：
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --output_model_path models/output_model.bin --encoder bert --target bert
The example of pre-training on single GPU (the id of GPU is 3)：
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --output_model_path models/output_model.bin --encoder bert --target bert --gpu_ranks 3
The example of pre-training on a single machine with 8 GPUs：
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--output_model_path models/output_model.bin --encoder bert --target bert --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7
The example of pre-training on two machines, each has 8 GPUs (16 GPUs in total):
Node-0 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--output_model_path models/output_model.bin --encoder bert --target bert --world_size 16 --gpu_ranks 0 1 2 3 4 5 6 7 \
--master_ip tcp://node-0-addr:port
Node-1 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--output_model_path models/output_model.bin --encoder bert --target bert --world_size 16 --gpu_ranks 8 9 10 11 12 13 14 15 \
--master_ip tcp://node-0-addr:port
We recommend to load a pre-trained model. We can specify the pre-trained model by --pretrained_model_path . The example of pre-training on CPU and single GPU:
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
--encoder bert --target bert
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
--encoder bert --target bert --gpu_ranks 3
The example of pre-training on a single machine with 8 GPUs：
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
--encoder bert --target bert --world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7
The example of pre-training on two machines, each has 8 GPUs (16 GPUs in total):
Node-0 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
--encoder bert --target bert --world_size 16 --gpu_ranks 0 1 2 3 4 5 6 7 --master_ip tcp://node-0-addr:port
Node-1 : python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt \
--pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
--encoder bert --target bert --world_size 16 --gpu_ranks 8 9 10 11 12 13 14 15 --master_ip tcp://node-0-addr:port
UER-py allows users to combine different components (e.g. subencoders, encoders, and targets). Here is an example of trying different targets:
In fact, NSP target and sentence-level reviews are incompatible to some extent. We could replace BERT target with MLM target on book review dataset:
python3 preprocess.py --corpus_path corpora/book_review.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt --processes_num 8 --target mlm
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --pretrained_model_path models/google_model.bin --output_model_path models/output_model.bin \
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 --total_steps 20000 --save_checkpoint_steps 5000 --encoder bert --target mlm
Notice that different targets correspond to different corpus formats. It is important to select proper format for a target. If we want to change encoder, only thing we need to do is to specify --encoder in pretrain.py. Here is an example of using LSTM for pre-training.
python3 preprocess.py --corpus_path corpora/book_review.txt --vocab_path models/google_vocab.txt --dataset_path dataset.pt --processes_num 8 --target lm
python3 pretrain.py --dataset_path dataset.pt --vocab_path models/google_vocab.txt --output_model_path models/output_model.bin \
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 --total_steps 20000 --save_checkpoint_steps 5000 \
--encoder lstm --target lm --learning_rate 1e-3 --config_path models/rnn_config.json
Currently, UER-py consists of 4 downstream tasks, i.e. classification, sequence labeling, cloze test, feature extractor. The encoder of downstream task should be coincident with the pre-trained model.
classifier.py adds two feedforward layers upon encoder layer.
usage: classifier.py [-h] [--pretrained_model_path PRETRAINED_MODEL_PATH]
[--output_model_path OUTPUT_MODEL_PATH]
[--vocab_path VOCAB_PATH] --train_path TRAIN_PATH
--dev_path DEV_PATH --test_path TEST_PATH
[--config_path CONFIG_PATH] [--batch_size BATCH_SIZE]
[--seq_length SEQ_LENGTH]
[--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
[--bidirectional] [--pooling {mean,max,first,last}]
[--subword_type {none,char}]
[--sub_vocab_path SUB_VOCAB_PATH]
[--subencoder {avg,lstm,gru,cnn}]
[--sub_layers_num SUB_LAYERS_NUM]
[--tokenizer {bert,char,word,space}]
[--learning_rate LEARNING_RATE] [--warmup WARMUP]
[--dropout DROPOUT] [--epochs_num EPOCHS_NUM]
[--report_steps REPORT_STEPS] [--seed SEED]
[--mean_reciprocal_rank]
The example of using classifier.py：
python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/book_review/train.tsv --dev_path datasets/book_review/dev.tsv --test_path datasets/book_review/test.tsv \
--epochs_num 3 --batch_size 64 --encoder bert
The example of using classifier.py for pair classification:
python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/lcqmc/train.tsv --dev_path datasets/lcqmc/dev.tsv --test_path datasets/lcqmc/test.tsv \
--epochs_num 3 --batch_size 64 --encoder bert
The example of using classifier.py for dbqa:
python3 classifier.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/dbqa/train.tsv --dev_path datasets/dbqa/dev.tsv --test_path datasets/dbqa/test.tsv \
--epochs_num 3 --batch_size 64 --encoder bert --mean_reciprocal_rank
tagger.py adds a feedforward layer upon encoder layer.
usage: tagger.py [-h] [--pretrained_model_path PRETRAINED_MODEL_PATH]
[--output_model_path OUTPUT_MODEL_PATH]
[--vocab_path VOCAB_PATH] [--train_path TRAIN_PATH]
[--dev_path DEV_PATH] [--test_path TEST_PATH]
[--config_path CONFIG_PATH] [--batch_size BATCH_SIZE]
[--seq_length SEQ_LENGTH]
[--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
[--bidirectional] [--subword_type {none,char}]
[--sub_vocab_path SUB_VOCAB_PATH]
[--subencoder {avg,lstm,gru,cnn}]
[--sub_layers_num SUB_LAYERS_NUM]
[--learning_rate LEARNING_RATE] [--warmup WARMUP]
[--dropout DROPOUT] [--epochs_num EPOCHS_NUM]
[--report_steps REPORT_STEPS] [--seed SEED]
The example of using tagger.py：
python3 tagger.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--train_path datasets/msra/train.tsv --dev_path datasets/msra/dev.tsv --test_path datasets/msra/test.tsv \
--epochs_num 5 --batch_size 32 --encoder bert
cloze.py predicts masked words. Top n words are returned.
usage: cloze.py [-h] [--pretrained_model_path PRETRAINED_MODEL_PATH]
[--vocab_path VOCAB_PATH] [--input_path INPUT_PATH]
[--output_path OUTPUT_PATH] [--config_path CONFIG_PATH]
[--batch_size BATCH_SIZE] [--seq_length SEQ_LENGTH]
[--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
[--bidirectional] [--target {bert,lm,cls,mlm,nsp,s2s}]
[--subword_type {none,char}] [--sub_vocab_path SUB_VOCAB_PATH]
[--subencoder_type {avg,lstm,gru,cnn}]
[--tokenizer {bert,char,word,space}] [--topn TOPN]
The example of using cloze.py：
python3 cloze.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--input_path datasets/cloze_input.txt --output_path output.txt
feature_extractor.py extracts sentence embeddings.
usage: feature_extractor.py [-h] --input_path INPUT_PATH --model_path
MODEL_PATH --vocab_path VOCAB_PATH --output_path
OUTPUT_PATH [--subword_type {none,char}]
[--sub_vocab_path SUB_VOCAB_PATH]
[--subencoder {avg,lstm,gru,cnn}]
[--sub_layers_num SUB_LAYERS_NUM]
[--seq_length SEQ_LENGTH]
[--batch_size BATCH_SIZE]
[--config_path CONFIG_PATH]
[--encoder {bert,lstm,gru,cnn,gatedcnn,attn,rcnn,crnn,gpt}]
[--target {bert,lm,cls,mlm,nsp,s2s}]
[--tokenizer {char,word,space,mixed}]
The example of using feature_extractor.py：
python3 feature_extractor.py --input_path datasets/cloze_input.txt --pretrained_model_path models/google_model.bin \
--vocab_path models/google_vocab.txt --output_path output.npy
Pre-trained models can learn high-quality word embeddings. Traditional word embeddings such as word2vec and GloVe assign each word a fixed vector. However, polysemy is a pervasive phenomenon in human language, and the meanings of a polysemous word depend on the context. To this end, we use a the hidden state in pre-trained models to represent a word. It is noticeable that Google BERT is a character-based model. To obtain real word embedding (not character embedding), Users should download our word-based BERT model and vocabulary. The example of using scripts/topn_words_indep.py (finding nearest neighbours for context-independent word embedding)：
python3 scripts/topn_words_indep.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--cand_vocab_path models/google_vocab.txt --target_words_path target_characters.txt
Contexct-independent word embedding is obtained model's embedding layer. The format of the target_characters.txt is as follows:
word-1
word-2
...
word-n
The example of using scripts/topn_words_dep.py (finding nearest neighbours for context-dependent word embedding)：
python3 scripts/topn_words_dep.py --pretrained_model_path models/google_model.bin --vocab_path models/google_vocab.txt \
--cand_vocab_path models/google_vocab.txt --sent_path target_characters_with_sentences.txt --config_path models/google_config.json \
--batch_size 256 --seq_length 32 --tokenizer bert
We substitute the target word with other words in the vocabulary and feed the sentences into the pretrained model. Hidden state is used as the context-dependent embedding of a word. Select proper tokenizer according to the sentence in target_characters_with_sentences.txt. The format of target_words_with_sentences.txt is as follows:
sent1 word1
sent1 word1
...
sentn wordn
Sentence and word are splitted by \t.
| Scripts | Function description |
|---|---|
| average_model.py | Take the average of pre-trained models. A frequently-used ensemble strategy for deep learning models |
| build_vocab.py | Build vocabulary (multi-processing supported) |
| check_model.py | Check the model (single GPU or multiple GPUs) |
| diff_vocab.py | Compare two vocabularies |
| dynamic_vocab_adapter.py | Change the pre-trained model according to the vocabulary. It can save memory in fine-tuning stage since task-specific vocabulary is much smaller than general-domain vocabulary |
| multi_single_convert.py | convert the model (single GPU or multiple GPUs) |
| topn_words_indep.py | Finding nearest neighbours with context-independent word embedding |
| topn_words_dep.py | Finding nearest neighbours with context-dependent word embedding |
GPU：Tesla P40
CPU：Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
We use BERT to test the speed of distributed training mode. Google BERT is trained for 1 million steps and each step contains 128,000 tokens. It takes around 18 days to reproduce the experiments by UER-py on 3 GPU machines (24 GPU in total).
| #(machine) | #(GPU)/machine | tokens/second |
|---|---|---|
| 1 | 0 | 276 |
| 1 | 1 | 7050 |
| 1 | 2 | 13071 |
| 1 | 4 | 24695 |
| 1 | 8 | 44300 |
| 3 | 8 | 84386 |
We qualitatively evaluate pre-trained models by finding words' near neighbours.
Evaluation of context-independent word embedding:
| Target word: 苹 | Target word: 吃 | Target word: 水 | |||
| 蘋 | 0.762 | 喝 | 0.539 | 河 | 0.286 |
| apple | 0.447 | 食 | 0.475 | 海 | 0.278 |
| iphone | 0.400 | 啃 | 0.340 | water | 0.276 |
| 柠 | 0.347 | 煮 | 0.324 | 油 | 0.266 |
| ios | 0.317 | 嚐 | 0.322 | 雨 | 0.259 |
Evaluation of context-dependent word embedding:
Target sentence: 其冲积而形成小平原沙土层厚而肥沃，盛产苹果、大樱桃、梨和葡萄。
| Target word: 苹 | |
| 蘋 | 0.822 |
| 莓 | 0.714 |
| 芒 | 0.706 |
| 柠 | 0.704 |
| 樱 | 0.696 |
Target sentence: 苹果削减了台式Mac产品线上的众多产品
| Target word: 苹 | |
| 蘋 | 0.892 |
| apple | 0.788 |
| iphone | 0.743 |
| ios | 0.720 |
| ipad | 0.706 |
Evaluation of context-independent word embedding:
| Target word: 苹果 | Target word: 腾讯 | Target word: 吉利 | |||
| 苹果公司 | 0.419 | 新浪 | 0.357 | 沃尔沃 | 0.277 |
| apple | 0.415 | 网易 | 0.356 | 伊利 | 0.243 |
| 苹果电脑 | 0.349 | 搜狐 | 0.356 | 长荣 | 0.235 |
| 微软 | 0.320 | 百度 | 0.341 | 天安 | 0.224 |
| mac | 0.298 | 乐视 | 0.332 | 哈达 | 0.220 |
Evaluation of context-dependent word embedding:
Target sentence: 其冲积而形成小平原沙土层厚而肥沃，盛产苹果、大樱桃、梨和葡萄。
| Target word: 苹果 | |
| 柠檬 | 0.734 |
| 草莓 | 0.725 |
| 荔枝 | 0.719 |
| 树林 | 0.697 |
| 牡丹 | 0.686 |
Target sentence: 苹果削减了台式Mac产品线上的众多产品
| Target word: 苹果 | |
| 苹果公司 | 0.836 |
| apple | 0.829 |
| 福特 | 0.796 |
| 微软 | 0.777 |
| 苹果电脑 | 0.773 |
Target sentence: 讨吉利是通过做民间习俗的吉祥事，或重现过去曾经得到好结果的行为，以求得好兆头。
| Target word: 吉利 | |
| 仁德 | 0.749 |
| 光彩 | 0.743 |
| 愉快 | 0.736 |
| 永元 | 0.736 |
| 仁和 | 0.732 |
Target sentence: 2010年6月2日福特汽车公司宣布出售旗下高端汽车沃尔沃予**浙江省的吉利汽车，同时将于2010年第四季停止旗下中阶房车品牌所有业务
| Target word: 吉利 | |
| 沃尔沃 | 0.771 |
| 卡比 | 0.751 |
| 永利 | 0.745 |
| 天安 | 0.741 |
| 仁和 | 0.741 |
Target sentence: 主要演员有扎克·布拉夫、萨拉·朝克、唐纳德·费森、尼尔·弗林、肯·詹金斯、约翰·麦吉利、朱迪·雷耶斯、迈克尔·莫斯利等。
| Target word: 吉利 | |
| 玛利 | 0.791 |
| 米格 | 0.768 |
| 韦利 | 0.767 |
| 马力 | 0.764 |
| 安吉 | 0.761 |
We use a range of Chinese datasets to evaluate the performance of UER-py. Douban book review, ChnSentiCorp, Shopping, and Tencentnews are sentence-level small-scale sentiment classification datasets. MSRA-NER is a sequence labeling dataset. These datasets are included in this project. Dianping, JDfull, JDbinary, Ifeng, and Chinanews are large-scale classification datasets. They are collected in glyph and can be downloaded at glyph's github project. These five datasets don't contain validation set. We use 10% instances in trainset for validation.
Most pre-training models consist of 2 stages: pre-training on general-domain corpus and fine-tuning on downstream dataset. We recommend 3-stage mode: 1)Pre-training on general-domain corpus; 2)Pre-training on downstream dataset; 3)Fine-tuning on downstream dataset. Stage 2 enables models to get familiar with distributions of downstream tasks. It is sometimes known as semi-supervised fune-tuning.
Hyper-parameter settings are as follows:
- Stage 1: We train with batch size of 256 sequences and each sequence contains 256 tokens. We load Google's pretrained models and train upon it for 500,000 steps. The learning rate is 2e-5 and other optimizer settings are identical with Google BERT. BERT tokenizer is used.
- Stage 2: We train with batch size of 256 sequences. For classification datasets, the sequence length is 128. For sequence labeling datasets, the sequence length is 256. We train upon Google's pretrained model for 20,000 steps. Optimizer settings and tokenizer are identical with stage 1.
- Stage 3: For classification datasets, the training batch size and epochs are 64 and 3. For sequence labeling datasets, the training batch size and epochs are 32 and 5. Optimizer settings and tokenizer are identical with stage 1.
We provide the pre-trained models (using BERT target) on different downstream datasets: book_review_model.bin; chnsenticorp_model.bin; shopping_model.bin; msra_model.bin. Tencentnews dataset and its pretrained model will be publicly available after data desensitization.
| Model/Dataset | Douban book review | ChnSentiCorp | Shopping | MSRA-NER | Tencentnews review |
|---|---|---|---|---|---|
| BERT | 87.5 | 94.3 | 96.3 | 93.0/92.4/92.7 | 84.2 |
| BERT+semi_BertTarget | 88.1 | 95.6 | 97.0 | 94.3/92.6/93.4 | 85.1 |
| BERT+semi_MlmTarget | 87.9 | 95.5 | 97.1 | 85.1 |
Pre-training is also important for other encoders and targets. We pre-train a 2-layer LSTM on 1.9G review corpus with language model target. Embedding size and hidden size are 512. The model is much more efficient than BERT in pre-training and fine-tuning stages. We show that pre-training brings significant improvements and achieves competitive results (the differences are not big compared with the results of BERT).
| Model/Dataset | Douban book review | ChnSentiCorp | Shopping |
|---|---|---|---|
| BERT | 87.5 | 94.3 | 96.3 |
| LSTM | 80.2 | 88.3 | 94.4 |
| LSTM+pre-training | 86.6(+6.4) | 94.5(+6.2) | 96.5(+2.1) |
It requires tremendous computional resources to fine-tune on large-scale datasets. For Ifeng, Chinanews, Dianping, JDbinary, and JDfull datasets, we provide their classification models (see Chinese model zoo). Classification models on large-scale datasets allow users to reproduce the results without training. Besides that, classification models could be used for improving other related tasks. More experimental results will come soon.
Ifeng and Chinanews datasets contain news' titles and abstracts. In stage 2, we use title to predict abstract.
| Model/Dataset | Ifeng | Chinanews | Dianping | JDbinary | JDfull |
|---|---|---|---|---|---|
| pre-SOTA (Glyph & Glyce) | 85.76 | 91.88 | 78.46 | 91.76 | 54.24 |
| BERT | 87.50 | 93.37 | 92.37 | 54.79 | |
| BERT+semi+BertTarget | 87.65 |
We also provide the pre-trained models on different corpora, encoders, and targets (see Chinese model zoo). Selecting proper pre-training models is beneficial to the performance of downstream tasks.
| Model/Dataset | MSRA-NER |
|---|---|
| Wikizh corpus (Google) | 93.0/92.4/92.7 |
| Renminribao corpus | 94.4/94.4/94.4 |
With the help of UER, we are pre-training models with different corpora, encoders, and targets.
| pre-trained model | Link | Description |
|---|---|---|
| Wikizh+BertEncoder+BertTarget | https://share.weiyun.com/5s9AsfQ | The training corpus is Wiki_zh, trained by Google |
| Wikizh(word-based)+BertEncoder+BertTarget | Model: https://share.weiyun.com/5s4HVMi Vocab: https://share.weiyun.com/5NWYbYn | Word-based BERT model trained on Wikizh. Training steps: 500,000 |
| RenMinRiBao+BertEncoder+BertTarget | https://share.weiyun.com/5JWVjSE | The training corpus is news data from People's Daily (1946-2017). It is suitable for datasets related with news, e.g. F1 is improved on MSRA-NER from 92.6 to 94.4 (compared with Google BERT). Training steps: 500,000 |
| Webqa2019+BertEncoder+BertTarget | https://share.weiyun.com/5HYbmBh | The training corpus is WebQA, which is suitable for datasets related with social media, e.g. Accuracy (dev/test) on LCQMC is improved from 88.8/87.0 to 89.6/87.4; Accuracy (dev/test) on XNLI is improved from 78.1/77.2 to 79.0/78.8 (compared with Google BERT). Training steps: 500,000 |
| Google-BERT-en-uncased-base | Model: https://share.weiyun.com/5hWivED Vocab: https://share.weiyun.com/5gBxBYD | Provided by Google. |
| Google-BERT-en-cased-base | Model: https://share.weiyun.com/5SltATz Vocab: https://share.weiyun.com/5ouUo2q | Provided by Google. |
| Reviews+LstmEncoder+LmTarget | https://share.weiyun.com/57dZhqo | The training corpus is amazon reviews + JDbinary reviews + dainping reviews (11.4M reviews in total). Language model target is used. It is suitable for datasets related with reviews. It achieves over 5 percent improvements on some review datasets compared with random initialization. Training steps: 200,000; Sequence length: 128 |
| (Mixedlarge corpus & Amazon reviews)+LstmEncoder+(LmTarget & ClsTarget) | https://share.weiyun.com/5B671Ik | Mixedlarge corpus contains baidubaike + wiki + webqa + RenMinRiBao. The model is trained on it with language model target. And then the model is trained on Amazon reviews with language model and classification targets. It is suitable for datasets related with reviews. It can achieve comparable results with BERT on some review datasets. Training steps: 500,000 + 100,000; Sequence length: 128 |
| IfengNews+BertEncoder+BertTarget | https://share.weiyun.com/5HVcUWO | The training corpus is news data from Ifeng website. We use news titles to predict news abstracts. Training steps: 100,000; Sequence length: 128 |
| jdbinary+BertEncoder+ClsTarget | https://share.weiyun.com/596k2bu | The training corpus is review data from JD (jingdong). Classification target is used for pre-training. It is suitable for datasets related with shopping reviews, e.g. accuracy is improved on shopping datasets from 96.3 to 97.2 (compared with Google BERT). Training steps: 50,000; Sequence length: 128 |
| jdfull+BertEncoder+MlmTarget | https://share.weiyun.com/5L6EkUF | The training corpus is review data from JD (jingdong). Masked LM target is used for pre-training. Training steps: 50,000; Sequence length: 128 |
| Amazonreview+BertEncoder+ClsTarget | https://share.weiyun.com/5XuxtFA | The training corpus is review data from Amazon (including book reviews, movie reviews, and etc.). Classification target is used for pre-training. It is suitable for datasets related with reviews, e.g. accuracy is improved on Douban book review datasets from 87.6 to 88.5 (compared with Google BERT). Training steps: 20,000; Sequence length: 128 |
| XNLI+BertEncoder+ClsTarget | https://share.weiyun.com/5oXPugA | Infersent with BertEncoder |
We release the classification models on 5 large-scale datasets, i.e. Ifeng, Chinanews, Dianping, JDbinary, and JDfull. Users can use these models to reproduce results, or regard them as pre-training models for other datasets.
| Datasets | Link |
|---|---|
| Ifeng | https://share.weiyun.com/5ZCp4wU |
| Chinanews | https://share.weiyun.com/5bSfeQ7 |
| Dianping | https://share.weiyun.com/5Ls8R02 |
| JDbinary | https://share.weiyun.com/5QNu4QF |
| JDfull | https://share.weiyun.com/5bqchN1 |
For communication related to this project, please contact Zhe Zhao (helloworld@ruc.edu.cn; nlpzhezhao@tencent.com) or Xin Zhao (2014201975@ruc.edu.cn).