/awesome-pruning

awesome-pruning

PRs WelcomeAwesome

Table of Contents

0. Overview

The repo includes the ongoing updates of representative neural network pruning papers and open-source codes.
Our paper [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations] (Paper Link) is under review.

Taxonomy: In our survey, we provide a comprehensive review of the state-of-the-art in deep neural network pruning, which we categorize along five orthogonal axes: Universal/Specific Speedup, When to Prune, Pruning Criteria, Learn to Prune, and Fusion of Pruning and Other Techniques.

1. When to Prune

1.1 Static Pruning

Type L F C N 'H' W P Other
Explanation Layer pruning Filter pruning Channel pruning Neuron pruning Head pruning Weight pruning Pioneer other types

1.1.1 Pruning Before Training

No. Title Venue Type Algorithm Name Code APP Year
01 SNIP: Single-shot Network Pruning based on Connection Sensitivity ICLR W&P SNIP TensorFLow(Author) Image Classification 2019
02 A Signal Propagation Perspective for Pruning Neural Networks at Initialization ICLR (Spotlight) W - TensorFLow(Author) Image Classification 2020
03 Picking Winning Tickets before Training by Preserving Gradient Flow) ICLR W GraSP PyTorch(Author) Image Classification 2020
04 Pruning from Scratch AAAI C - PyTorch(Author) Image Classification 2020
05 Pruning neural networks without any data by iteratively conserving synaptic flow NeurIPS W SynFlow PyTorch(Author) Image Classification 2020
06 A Unified Paths Perspective for Pruning at Initialization arXiv W - - Image Classification 2021
07 Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot NeurIPS W Smart-Ratios PyTorch(Author) Image Classification 2020
08 Progressive Skeletonization: Trimming More Fat from a network at initialization ICLR W FORCE PyTorch(Author) Image Classification 2021
09 Robust Pruning at Initialization ICLR W SPB - Image Classification 2021
10 Prunining via Iterative Ranking of Sensitivity Statics arXiv WFC SNIP-it PyTorch(Author) Image Classification 2020
11 Prunining Neural Networks at Initialization: Why are We Missing the Mark? ICLR W - - Image Classification 2021
12 Why is Pruning at Initialization Immune to Reinitializating and Shuffling?) arXiv W - - Image Classification 2021
13 Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients ICLR WF ProsPr PyTorch(Author) Image Classification 2022
14 Dual Lottery Ticket Hypothesis ICLR W RST PyTorch(Author) Image Classification 2022
15 Recent Advances on Neural Network Pruning at Initialization IJCAI W - PyTorch(Author) Image Classification 2022
16 What’s Hidden in a Randomly Weighted Neural Network? CVPR W - PyTorch(Author) Image Classification 2020
17 Finding trainable sparse networks through Neural Tangent Transfer ICML W - PyTorch(Author) Image Classification 2020
18 The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training ICLR W - PyTorch(Author) Image Classification 2022

1.1.2 Pruning During Training

No. Title Venue Type Algorithm Name Code APP Year
01 Dynamic Sparse Training: Find Effective Sparse Network from Scratch with Trainable Masked Layers ICLR NF DST PyTorch(Author) Image Classification 2020
02 Learning Structured Sparsity in Deep Neural Networks NIPS FC SSL Caffe(Author) Image Classification 2016
03 Learning Efficient Convolutional Networks through Networks Slimming ICCV C Slimming Lua(Author) Image Classification 2017
04 Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers ICLR F - TensorFlow(Author) PyTorch(3rd) Image Classification&Segmentation 2018
05 Data-Driven Sparse Structure Selection for Deep Neural Networks ECCV F SSS MXNet(Author) Image Classification 2018
06 Compressing Convolutional Neural Networks via Factorized Convolutional Filters CVPR F FCF PyTorch(Author) Image Classification 2019
07 MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks CVPR L MorphNet PyTorch(Author) Image Classification 2018
08 Learning the Number of Neurons in Deep Networks NIPS N - - Image Classification 2016
09 Learning Sparse Neural Networks Through $L_0$ Regularization ICLR FN - PyTorch(Author) Image Classification 2018
10 Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks IJCAI F SFP PyTorch(Author) Image Classification 2018
11 Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration CVPR F FPGM PyTorch(Author) Image Classification 2019
12 Variational Convolutional Neural Network Pruning CVPR F VCP - Image Classification 2019
13 Rigging the Lottery:Making All Tickets Winners ICML W RigL PyTorch(Author) Image Classification 2019
14 NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm arXiv N NeST - Image Classification 2019
15 Sparse Training via Boosting Pruning Plasticity with Neuroregeneration NeurIPS WF GraNet PyTorch(Author) Image Classification 2021
16 DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation ECCV F DSA PyTorch(Author) Image Classification 2020
17 Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization ICML W DSR PyTorch(Not Available) Image Classification 2019
18 Sparse Networks from Scratch: Faster Training without Losing Performance arXiv W SM PyTorch(Author) Image Classification 2019
19 Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science Nature Communication W&P SET - Image Classification 2018
20 Online Filter Clustering and Pruning for Efficient Convets arXiv W - - Image Classification 2019
21 Dynamic Model Pruning with Feedback ICLR WF DPF PyTorch(3rd) Image Classification 2020
22 Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training ICML W ITOP PyTorch(Anthor) Image Classification 2021
23 Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset ICML W DCTpS PyTorch(Anthor) Image Classification 2021
24 Selfish Sparse RNN Training ICML W SNT-ASGD PyTorch(Anthor) Language Modeling 2021
25 Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity ICLR W FreeTickets PyTorch(Anthor) Image Classification 2022
26 Training Adversarially Robust Sparse Networks via Bayesian Connectivity Sampling ICML W - PyTorch(Anthor) Adversarial Robustness 2021
27 Dynamic Sparse Training for Deep Reinforcement Learning IJCAI W - PyTorch(Anthor) Continuous Control 2022
28 The State of Sparse Training in Deep Reinforcement Learning. ICML W - Tensorflow(Anthor) Continuous Control 2022
29 MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning ICCV F MetaPruning PyTorch(Author) Image Classification 2019
30 DHP: Differentiable Meta Pruning via HyperNetworks ECCV F DHP PyTorch(Author) Image Classification&Super-resolution&Denoising 2019
31 Global Sparse Momentum SGD for Pruning Very Deep Neural Networks NeurIPS W GSM PyTorch(Author) Image Classification 2019
32 Pruning Filter in Filter NeurIPS Other SWP PyTorch(Author) Image Classification 2020
33 Network Pruning via Transformable Architecture Search NeurIPS F TAS PyTorch(Author) Image Classification 2019
34 SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning ECCV W SuperTickets PyTorch(Author) Image Classification&Object Detection&Human Pose Estimation 2022
35 Exploring Sparsity in recurrent neural networks ICLR W - PyTorch Speech Recognition 2017
36 Training Neural Networks with Fixed Sparse Masks NeurIPS W - PyTorch(Author) Image Classification 2021
37 Deep Rewiring: Training very Sparse Deep Networks ICLR W - - Image Classification&Audio 2018

1.1.3 Pruning After Training

No. Title Venue Type Algorithm Name Code APP Year
01 Towards Optimal Structured CNN Pruning via Generative Adversarial Learning CVPR F GAL PyTorch(Author) Image Classification 2019
02 Winning the Lottery with Continuous Sparsification NeurIPS F CS PyTorch(Author) Image Classification 2019
03 Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure CVPR F C-SGD Tensorflow(Author) Image Classification 2019
04 ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression ICCV&TPAMI F ThiNet Caffe(Author), PyTorch(3rd) Image Classification 2017&2019
05 Channel pruning for accelerating very deep neural networks ICCV C - Caffe(Author) Image Classification&Object Detection 2017
06 NISP: Pruning Networks using Neuron Importance Score Propagation CVPR NC NISP - Image Classification 2018
07 Pruning Convolutional Neural Networks for Resource Efficient Inference ICLR F - PyTorch Image Classification 2017
08 Discrimination-aware Channel Pruning for Deep Neural Networks NeurIPS C DCP TensorFlow(Author) Image Classification 2018
09 Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks NeurIPS F Gate Decorator PyTorch(Author) Image Classification&Semantic Segmentation 2019
10 Pruning Filters for Efficient ConvNets ICLR F PFEC PyTorch(3rd) Image Classification 2017
11 Neural Network Pruning with Residual-Connections and Limited-Data CVPR C CURL PyTorch(Author) Image Classification 2020
12 HRank: Filter Pruning using High-Rank Feature Map CVPR F HRank Pytorch(Author) Image Classification 2020
13 Importance Estimation for Neural Network Pruning CVPR F Taylor-FO-BN PyTorch(Author) Image Classification 2019
14 Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework ICML F - - Image Classification 2021
15 Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration CVPR F LFPC - Image Classification 2020
16 Neural Pruning via Growing Regularization ICLR WF Greg - Image Classification 2021
17 Trainability Preserving Nueral Structured Pruning ICLR F TPP Pytorch(Author) Image Classification 2023
18 Optimal Brain Damage NIPS W OBD - Image Classification 1989
19 Second Order Derivatives for Network Pruning: Optimal Brain Surgeon NIPS W OBS - Image Classification 1992
20 Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding ICLR (Best) W - Caffe(Author) Image Classification 2016
21 The State of Sparsity in Deep Neural Networks arXiv w - TensorFlow(Author) Image Classification&machine translation 2019
22 Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks AAAI F - - Image Classification 2019
23 Reborn filters: Pruning convolutional neural networks with limited data AAAI F - - Image Classification 2020
24 ICLR F - PyTorch(Author) Image Classification 2021
25 Lottery Jackpot Exist in Pre-trained Models arXiv W Jackpot [PyTorch(Author)]（https://github.com/zyxxmu/lottery-jackpots） Image Classification 2021
26 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy AAAI W 2PFPCE - Image Classification 2018
27 Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression CVPR W KSE PyTorch(Author) Image Classification 2019
28 AMC: Automl for model compression and acceleration on mobile devices ECCV F AMC TensorFlow(3rd) Image Classification 2018
29 Towards Efficient Model Compression via Learned Global Ranking CVPR F LeGR Pytorch(Author) Image Classification 2020
30 Collaborative Channel Pruning for Deep Networks ICML F CCP - Image Classification 2019
31 ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model CVPR F ECC Pytorch(Author) Image Classification&Semantic Segmentation 2019
32 Discrete Model Compression With Resource Constraint for Deep Neural Networks CVPR F - - Image Classification 2020
33 Network Pruning via Performance Maximization CVPR F NPPM Pytorch(Author) Image Classification 2021
34 Operation-Aware Soft Channel Pruning using Differentiable Masks ICML F SCP - Image Classification 2020
35 Towards Compact and Robust Deep Networks arXiv W - - Image Classification 2020
36 HYDRA: Pruning Adversarially Robust Neural Networks NeurIPS W HYDRA PyTorch(Author) Adversarial Robustness 2020
37 Approximated Oracle Filter Pruning for Destructive CNN Width Optimization github ICML F AOFP Pytorch(Author) Image Classification 2019
38 Channel Pruning via Automatic Structure Search IJCAI F ABC PyTorch(Author) Image Classification 2020
39 Group Fisher Pruning for Practical Network Compression ICML F GFP PyTorch(Author) Image Classification&Object Detection 2021
40 TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning AAAI F TransTailor - Image Classification 2021
41 Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning TNNLS F SSR Caffe(Author) Image Classification 2019
42 Network Pruning That Matters: A Case Study on Retraining Variants ICLR F - PyTorch(Author) Image Classification 2021
43 ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations ICLR F ChipNet PyTorch(Author) Image Classification 2021
44 SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning ICLR (Spotlight) F SOSP PyTorch(Author)(Releasing) Image Classification 2022
45 The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks ICLR (Best) W LTH TensorFlow(Author) Image Classification 2019
46 Proving the Lottery Ticket Hypothesis for Convolutional Neural Networks ICML N - - - 2020
47 Logarithmic Pruning is All You Need NeurIPS N - - - 2020
48 Optimal Lottery Tickets via SUBSETSUM:Logarithmic Over-Parameterization is Sufficient NeurIPS N - PyTorch(Author) Image Classification 2020
49 Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot? NeurIPS W - PyTorch(Author) Image Classification 2021
50 Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network ICLR W MPTs PyTorch(Author) Image Classification 2021
51 One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers NeurIPS W - - Image Classification 2019
52 Long live the lottery: the existence of winning tickets in lifelong learning ICLR W - PyTorch(Author) Image Classification 2021
53 A Unified Lottery Ticket Hypothesis for Graph Neural Networks ICML W - PyTorch(Author) Node Classification&Link Prediction 2021
54 The Lottery Ticket Hypothesis for Pre-trained BERT Networks ICML W - PyTorch(Author) Language Modeling 2021
55 When BERT Plays the Lottery, All Tickets Are Winning EMNLP W - PyTorch(Author) Language Modeling 2020
56 Playing the Lottery with Rewards and Multiple Languages: Lottery Tickets in RL and NLP ICLR W - - Classic Control&Atari Game 2020
57 Playing Lottery Tickets with Vision and Language AAAI W - - Vision-and-Language 2022
58 Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory NeurIPS W - - Image Classification 2021
59 Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask NeurIPS W - TensorFlow(Author) Image Classification 2019
60 Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win AAAI W - PyTorch(Author) Image Classification 2022
61 Sparse Transfer Learning via Winning Lottery Tickets arXiv W - PyTorch(Author) Image Classification 2019
62 How many winning tickets are there in one DNN? arXiv W - - Image Classification 2020
63 CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-Quantization CVPR W CLIP-Q - Image Classification 2018
64 Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression CVPR F Hinge PyTorch(Author) Image Classification 2020
65 Towards Compact CNNs via Collaborative Compression CVPR F CC PyTorch(Author) Image Classification 2021
66 NPAS: A Compiler-aware Framework of Unified Network Pruning andArchitecture Search for Beyond Real-Time Mobile Acceleration CVPR F NPAS - Image Classification 2021
67 GAN Compression: Efficient Architectures for Interactive Conditional GANs arXiv C - - Image-to-Image Translation 2021
68 Content-Aware GAN Compression CVPR F - PyTorch(Author) Image Generation, Image Projection, Image Editing
69 Dreaming to Prune Image Deraining Networks CVPR F - - Image Deraining 2022
70 Prune Your Model Before Distill It ECCV F - PyTorch(Author) Image Classification 2022
71 LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression COLING W - - NLP(Sentiment Classification,Natural Language Inference,Pairwise Semantic Equivalence) 2020
72 The Lottery Ticket Hypothesis for Object Recognition CVPR W - PyTorch(Author) Object Detection 2021
73 Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian Relaxation IJCAI W - - Image Classification & Object Detection 2021
74 Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation CVPR F Joint-DetNAS - Image Classification & Object Detection 2021
75 Exploring Linear Relationship in Feature Map Subspace for ConvNets Compression arXiv F - - Object Detection&Human Pose Estimation 2018
76 Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers ICML W - - NLP 2020
77 To prune, or not to prune: exploring the efficacy of pruning for model compression ICLRW W - TensorFlow(Author) NLP 2018
78 Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned ACL W - PyTorch(Author) NLP 2019
79 Towards Adversarial Robustness Via Compact Feature Representations ICASSP N - PyTorch(Author) Adversarial Robustness 2021
80 Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior EMNLP Other - - NLP 2020
81 Reweighted Proximal Pruning for Large-Scale Language Representation arXiv Other - - NLP 2019
82 Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning EMNLP Other - - NLP 2019
83 EarlyBERT: Efficient BERT training via early-bird lottery tickets ACL-IJCNLP Other EarlyBERT PyTorch(Author) NLP 2021
84 Movement Pruning: Adaptive Sparsity by Fine-Tuning NeurIPS W - PyTorch(Author) NLP 2020
85 Audio Lottery: Speech Recognition made ultra-lightweight, transferable, and noise-robust ICLR W - PyTorch(Author) Speach Recognition 2022
86 PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition NeurIPS W PARP - Speach Recognition 2021
87 Dynamic Sparsity Neural Networks for Automatic Speech Recognition ICASSP W - - Speach Recognition 2021
88 On the Predictability of Pruning Across Scales ICML W - - Image Classification 2021
89 How much pre-training is enough to discover a good subnetwork? arXiv W - - Image Classification 2021
90 On the Transferability of Winning Tickets in Non-Natural Image Datasets arXiv W - - Image Classification 2020
91 The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models CVPR W - PyTorch(Author) Image Classification 2021
92 DepGraph: Towards Any Structural Pruning CVPR DepGraph - PyTorch(Author) CV/NLP 2023
93 How Well Do Sparse ImageNet Models Transfer? CVPR W - PyTorch(Author) Image Classification&Object Detection 2022
94 The Elastic Lottery Ticket Hypothesis NeurIPS W E-LTH PyTorch(Author) Image Classification 2021
95 Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Sparse Neural Networks NeurIPS W - - Image Classification 2021
96 Exploring Lottery Ticket Hypothesis in Spiking Neural Networks ECCV W ET PyTorch(Author) Image Classification 2022
97 Graph Pruning for Model Compression Applied Intelligence W GraphPruning - Image Classification 2022
98 Accelerating Convolutional Networks via Global & Dynamic Filter Pruning IJCAI F GDP - Image Classification 2018
99 DMCP: Differentiable Markov Channel Pruning for Neural Networks CVPR C DMCP - Image Classification 2020
100 Advancing Model Pruning via Bi-level Optimization NeurIPS WC BiP PyTorch(Author) Image Classification 2022
101 Structured Pruning Learns Compact and Accurate Models ACL LH CoFi PyTorch(Author) Natural Language Understanding 2022
102 The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models EMNLP W oBERT PyTorch(Author) Natural Language Understanding 2022
103 Lottery Jackpots Exist in Pre-trained Models arXiv W oBERT PyTorch(Author) Image Classification 2022
1.1.3.1 Post Training
No. Title Venue Type Algorithm Name Code APP Year
01 A Fast Post-Training Pruning Framework for Transformers NeurIPS HF - PyTorch(Author) Natural Language Understanding 2022
1.1.3.2 Pruning LLMs
No. Title Venue Type Algorithm Name Code APP Year
01 SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot NeurIPS W - PyTorch(Author) Language Modeling 2023
02 Pruning Meets Low-Rank Parameter-efficient arXiv W LoRAPrune - Image Classification&Language Modeling 2023
03 LLM-Pruner: On the Structural Pruning of Large Language Models arXiv LHP LLM-Pruner PyTorch(Author) Language Modeling 2023
04 Parameter-Efficient Sparsity for Large Language Models Fine-Tuning arXiv W PST PyTorch(Author) Language Modeling 2022

1.1.4 Pruning in Early Training

No. Title Venue Type Algorithm Name Code APP Year
01 Linear Mode Connectivity and the Lottery Ticket Hypothesis ICML W - - Image Classification 2020
02 When To Prune? A Policy Towards Early Structural Pruning CVPR F PaT - Image Classification 2022
03 Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks ICLR W - PyTorch(Author) Image Classification 2020

1.2 Dynamic Pruning

No. Title Venue Type Algorithm Name Code APP Year
01 Channel Gating Neural Networks NeurIPS F RNP - Image Classification 2017
02 Channel Gating Neural Networks NeurIPS C CGNet PyTorch(Author) Image Classification 2019
03 Dynamic Dual Gating Neural Networks ICCV C DGNet PyTorch(Author) Image Classification 2021
04 Manifold Regularized Dynamic Network Pruning CVPR F ManiDP PyTorch(Author) Image Classification 2021
05 Dynamic Channel Pruning: Feature Boosting and Suppression ICLR C FBS PyTorch(Author) Image Classification 2019
06 Frequency-Domain Dynamic Pruning for Convolutional Neural Networks NeurIPS F FDNP - Image Classification 2019
07 Fire Together Wire Together: A Dynamic Pruning Approach With Self-Supervised Mask Prediction CVPR F - - Image Classification 2019
08 Contrastive Dual Gating: Learning Sparse Features With Contrastive Learning CVPR WF CDG - Image Classification 2022

2. Learning and Pruning

2.1 Continual learning

No. Title Venue Algorithm Name Code APP Year
01 Learning Bayesian Sparse Networks With Full Experience Replay for Continual Learning CVPR SNCL - Image Classification 2022
02 Continual Prune-and-Select: Class-Incremental Learning with SPecialized Subnetworks Applied Intelligence - PyTorch(Author) Image Classification 2023

2.2 Contrastive learning

No. Title Venue Algorithm Name Code APP Year
01 Studying the impact of magnitude pruning on contrastive learning methods ICML - PyTorch(Author) Image Classification 2020

2.3 Federated learning

No. Title Venue Algorithm Name Code APP Year
01 FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server IJCAI FedDUAP - Image Classification 2020
02 Model Pruning Enables Efficient Federated Learning on Edge Devices TNNLS - PyTorch(Author) Image Classification 2022

3. Application

3.1 Computer Vision

No. Title Venue Code APP Year
01 SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning ECCV PyTorch(Author) Image Classification&Object Detection&Human Pose Estimation 2022
02 Training Neural Networks with Fixed Sparse Masks NeurIPS PyTorch(Author) Image Classification 2021
03 Deep Rewiring: Training very Sparse Deep Networks ICLR - Image Classification&Audio 2018
04 Co-Evolutionary Compression for Unpaired Image Translation ICCV PyTorch(Author) Image Style Translation 2019
05 Content-Aware GAN Compression CVPR PyTorch(Author) Image Style Translation 2021
06 Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space CVPR PyTorch(Author) Image Classification&Audio 2022

3.2 Natural Language Processing

No. Title Venue Code APP Year
01 A Fast Post-Training Pruning Framework for Transformers NeurIPS PyTorch(Author) Natural Language Understanding 2022
02 The Lottery Ticket Hypothesis for Pre-trained BERT Networks ICML PyTorch(Author) Language Modeling 2021
03 When BERT Plays the Lottery, All Tickets Are Winning EMNLP PyTorch(Author) Language Modeling 2020
04 Structured Pruning Learns Compact and Accurate Models ACL PyTorch(Author) Natural Language Understanding 2022
05 A Fast Post-Training Pruning Framework for Transformers NeurIPS PyTorch(Author) Natural Language Understanding 2022
06 The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models EMNLP PyTorch(Author) Natural Language Understanding 2022
07 Pruning Meets Low-Rank Parameter-efficient arXiv - Image Classification&Language Modeling 2023
08 LLM-Pruner: On the Structural Pruning of Large Language Models arXiv - Language Modeling 2023

3.3 Audio Signal Processing

No. Title Venue Code APP Year
01 Exploring Sparsity in recurrent neural networks ICLR PyTorch Speech Recognition 2017
02 Deep Rewiring: Training very Sparse Deep Networks ICLR - Image Classification&Audio 2018

4. Combination

4.1 Pruning and Quantization

No. Title Venue Code APP Year
01 Accelerating Sparse Deep Neural Networks arXiv - Image Classification&Object Detection&Language Translation&Language Modeling&Image Synthesis&Domain Translation&Style Transfer&Image-Image Translation&Super Resolution 2021
02 LLM-Pruner: On the Structural Pruning of Large Language Models arXiv PyTorch Causal Language Modeling 2023
03 Deep Model Compression Based on the Training History arXiv - Image Classification 2022
04 OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization AAAI - Image Classification 2021

5. Survey of Pruning

No. Title Venue Code APP Year
01 Pruning Algorithms-A Survey IEEE Transactions on Neural Networks - Image Classification 1993
02 Efficient Processing of Deep Neural Networks: A Tutorial and Survey arXiv - Image Classification 2017
03 Recent advances in efficient computation of deep convolutional neural networks arXiv - - 2018
04 The State of Sparsity in Deep Neural Networks arXiv PyTorch(Author) Image Classification&machine translation 2019
05 Convolutional Neural Network Pruning: A Survey CCC - - 2020
06 What is the State of Neural Network Pruning? MLSys - - 2020
07 A comprehensive survey on model compression and acceleration Artificial Intelligence Review - - 2020
08 A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions arXiv - - 2020
09 A Survey of Model Compression and Acceleration for Deep Neural Networks arXiv - - 2020
10 An Survey of Neural Network Compression arXiv - - 2020
11 Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey IEEE - - 2020
12 Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Survey arXiv - Image Classification 2020
13 Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks JMLR - Image Classification 2021
14 Dynamic Neural Networks: A Survey arXiv - - 2021
15 Pruning and Quantization for Deep Neural Network Acceleration: A Survey Neurocomputing - Image Classification 2021
16 Recent Advances on Neural Network Pruning at Initialization IJCAI - CV&NLP 2022
17 A Survey on Efficient Convolutional Neural Networks and Hardware Acceleration Electronics - - 2022
18 Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey arXiv - Image Classification 2022
19 A Survey on Dynamic Neural Networks for Natural Language Processing arXiv - NLP 2023
20 Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning arXiv PyTorch(Author) Image Classification 2023
21 Structured Pruning for Deep Convolutional Neural Networks: A survey arXiv - CV&NLP 2023
22 Transforming Large-Size to Lightweight Deep Neural Networks for IoT Applications ACM Computing Surveys - CV&NLP&Audio 2023

6. Other Works

No. Title Venue Algorithm Name Code APP Year
01 Are All Layers Created Equal? JMLR - - Image Classification 2022
02 Is Pruning Compression?: Investigating Pruning Via Network Layer Similarity WACV - - Image Classification 2020
03 A Gradient Flow Framework For Analyzing Network Pruning ICLR - PyTorch(Author) Image Classification 2021

Other Useful Links

https://github.com/airaria/TextPruner

Acknowledgements

We would like to express our gratitude to the authors of the articles cited in our survey and the authors of the following repositories.

https://github.com/he-y/awesome-Pruning/
https://github.com/MingSun-Tse/Awesome-Pruning-at-Initialization
https://github.com/csyhhu/Awesome-Deep-Neural-Network-Compression/blob/master/Paper/Pruning.md

Citation

If you find this project useful, please cite

@article{cheng2023survey,
  title={A Survey on Deep Neural Network Pruning:Taxonomy, Comparison, Analysis, and Recommendations},
  author={Hongrong Cheng and Miao Zhang and Javen Qinfeng Shi},
  journal={arXiv preprint arXiv:2308.06767},
  year={2023}
}