Must-read papers on improving efficiency for pre-trained language models.
The paper list is mainly mantained by Lei Li.
-
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter NeurIPS workshop
Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf [pdf] [project]
-
Patient Knowledge Distillation for BERT Model Compression EMNLP 2019
-
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models Preprint
Iulia Turc, Ming-Wei Chang, Kenton Lee, Kristina Toutanova [pdf] [project]
-
TinyBERT: Distilling BERT for Natural Language Understanding Findings of EMNLP 2020
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu [pdf] [project]
-
DynaBERT: Dynamic BERT with Adaptive Width and Depth NeurIPS 2020
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu [pdf] [project]
-
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing EMNLP 2020
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou [pdf] [project]
-
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers NeurIPS 2020
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou [pdf] [project]
-
BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance EMNLP 2020
Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, Yaohong Jin [pdf] [project]
-
MixKD: Towards Efficient Distillation of Large-scale Language Models ICLR 2021
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, Lawrence Carin [pdf]
-
Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains ACL-IJCNLP 2021
Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, Jun Huang [pdf]
-
MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillation ACL-IJCNLP 2021
Ahmad Rashid, Vasileios Lioutas, Mehdi Rezagholizadeh [pdf]
-
Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor ACL-IJCNLP 2021
Xinyu Wang, Yong Jiang, Zhaohui Yan, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu [pdf] [project]
-
Weight Distillation: Transferring the Knowledge in Neural Network Parameters ACL-IJCNLP 2021
Ye Lin, Yanyang Li, Ziyang Wang, Bei Li, Quan Du, Tong Xiao, Jingbo Zhu [pdf]
-
Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation ACL-IJCNLP 2021
Yuanxin Liu, Fandong Meng, Zheng Lin, Weiping Wang, Jie Zhou [pdf]
-
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers Findings of ACL-IJCNLP 2021
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei [pdf] [project]
-
One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers Findings of ACL-IJCNLP 2021
Chuhan Wu, Fangzhao Wu, Yongfeng Huang [pdf]
-
Dynamic Knowledge Distllation for Pre-trained Language Models EMNLP 2021
Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun [pdf] [project]
-
EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation Findings of EMNLP 2021
Chenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng, Xiaozhe Ren, Xiaodan Liang [pdf] [project]
-
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression EMNLP 2021
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei [pdf] [project]
-
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding Findings of EMNLP 2021
Tianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma, Ali Ghodsi, Mehdi Rezagholizadeh [pdf]
-
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers EMNLP 2020
Yimeng Wu, Peyman Passban, Mehdi Rezagholizadeh, Qun Liu [pdf] [project]
-
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation AAAI 2021
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, Qun Liu [pdf]
-
Universal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation EMNLP 2021
Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, Ali Ghodsi [pdf]
-
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference ACL 2020
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, Jimmy Lin [pdf] [project]
-
FastBERT: a Self-distilling BERT with Adaptive Inference Time ACL 2020
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, Qi Ju [pdf] [project]
-
The Right Tool for the Job: Matching Model and Instance Complexities ACL 2020
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, Noah A. Smith [pdf] [project]
-
A Global Past-Future Early Exit Method for Accelerating Inference of Pre-trained Language Models NAACL 2021
Kaiyuan Liao, Yi Zhang, Xuancheng Ren, Qi Su, Xu Sun, Bin He [pdf] [project]
-
CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade Preprint
Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun [pdf] [project]
-
Early Exiting BERT for Efficient Document Ranking SustaiNLP 2020
Ji Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin [pdf] [project]
-
BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression EACL 2021
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin [pdf] [project]
-
Accelerating BERT Inference for Sequence Labeling via Early-Exit ACL 2021
Xiaonan Li, Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, Xuanjing Huang [pdf] [project]
-
BERT Loses Patience: Fast and Robust Inference with Early Exit NeurIPS 2020
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, Furu Wei [pdf] [project]
-
Early Exiting with Ensemble Internal Classifiers Preprint
Tianxiang Sun, Yunhua Zhou, Xiangyang Liu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, Xipeng Qiu [pdf]
-
LeeBERT: Learned Early Exit for BERT with Cross-Level Optimization ACL 2021
Wei Zhu [pdf]
-
Consistent Accelerated Inference via Confident Adaptive Transformers EMNLP 2021
Tal Schuster, Adam Fisch, Tommi Jaakkola, Regina Barzilay [pdf] [project]
-
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline NAACL 2022
Xiangyang Liu, Tianxiang Sun, Junliang He, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, Xipeng Qiu [pdf] [project]
-
A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation Findings of ACL 2022
Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, Xipeng Qiu [pdf]
-
SkipBERT: Efficient Inference with Shallow Layer Skipping ACL 2022
Jue Wang, Ke Chen, Gang Chen, Lidan Shou, Julian McAuley [pdf] [project]
-
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT AAAI 2020
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, Kurt Keutzer [pdf] [project]
-
TernaryBERT: Distillation-aware Ultra-low Bit BERT EMNLP 2020
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, Qun Liu [pdf] [project]
-
Q8BERT: Quantized 8Bit BERT NeurIPS 2019 Workshop
Ofir Zafrir, Guy Boudoukh, Peter Izsak, Moshe Wasserblat [pdf] [project]
-
BinaryBERT: Pushing the Limit of BERT Quantization EMNLP 2020
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, Irwin King [pdf] [project]
-
Automatic Mixed-Precision Quantization Search of BERT IJCAI 2021
Changsheng Zhao, Ting Hua, Yilin Shen, Qian Lou, Hongxia Jin [pdf]
-
I-BERT: Integer-only BERT Quantization ICML 2021
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, Kurt Keutzer [pdf] [project]
-
Training with Quatization Noise for Extreme Model CompressionN ICLR 2021
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval Herve Jegou, Armand Joulin [pdf] [project]
-
Compression of Generative Pre-trained Language Models via Quantization ACL 2022
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, Ngai Wong [pdf]
-
BiBERT: Accurate Fully Binarized BERT ICLR 2022
Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua YAN, Aishan Liu, Qingqing Dang, Ziwei Liu, Xianglong Liu [pdf]
-
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned ACL 2019
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, Ivan Titov [pdf] [project]
-
Are Sixteen Heads Really Better than One? NeurIPS 2019
-
The Lottery Ticket Hypothesis for Pre-trained BERT Networks NeurIPS 2020
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin [pdf] [project]
-
Movement Pruning: Adaptive Sparsity by Fine-Tuning NeurIPS 2020
-
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning Rep4NLP 2020
Mitchell A. Gordon, Kevin Duh, Nicholas Andrews [pdf] [project]
-
Reducing Transformer Depth on Demand with Structured Dropout Preprint
Angela Fan, Edouard Grave, Armand Joulin [pdf]
-
When BERT Plays the Lottery, All Tickets Are Winning EMNLP 2020
-
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior Findings of EMNLP 2020
Zi Lin, Jeremiah Liu, Zi Yang, Nan Hua, Dan Roth [pdf]
-
Structured Pruning of a BERT-based Question Answering Model Preprint
J.S. McCarley, Rishav Chakravarti, Avirup Sil [pdf]
-
Structured Pruning of Large Language Models EMNLP 2020
-
Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm NAACL 2021
Dongkuan Xu, Ian E.H. Yen, Jinxi Zhao, Zhibin Xiao [pdf]
-
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization ACL 2021
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen [pdf] [project]
-
Block Pruning For Faster Transformers EMNLP 2021
François Lagunas, Ella Charlaix, Victor Sanh, Alexander M. Rush [pdf] [project]
-
A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models NeurIPS 2022
Yuanxin Liu, Fandong Meng, Zheng Lin, Jiangnan Li, Peng Fu, Yanan Cao, Weiping Wang, Jie Zhou [pdf]
-
Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection arxiv 2019
Guangxiang Zhao, Junyang Lin, Zhiyuan Zhang, Xuancheng Ren, Qi Su, Xu Sun [pdf] [project]
-
Compressing Pre-trained Language Models by Matrix Decomposition AACL 2020
Matan Ben Noach, Yoav Goldberg [pdf]
-
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression COLING 2020
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, Jing Bai [pdf]
-
Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators ACL 2021
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Zhi-Yuan Xie, Zhong-Yi Lu and Ji-Rong Wen [pdf] [project]
-
Exploring Extreme Parameter Compression for Pre-trained Language Models ICLR 2022
Benyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang, Qun Liu [pdf] [project]
-
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models Findings of EMNLP 2022
Lei Li, Yankai Lin, Xuancheng Ren, Guangxiang Zhao, Peng Li, Jie Zhou, Xu Sun [pdf] [project]
If you find any related work not included in the list, do not hesitate to raise a PR to help us complete the list.