/laasp-filter-pruning

Deepak Ghimire, Kilho Lee, and Seong-heum Kim, Loss-aware automatic selection of structured pruning criteria for deep neural network acceleration, Image and Vision Computing, vol. 136, p. 104745, 2023.

Primary LanguagePython

LAASP: Loss-Aware Automatic Selection of Filter Pruning Criteria for Deep Neural Network Acceleration

alt text

Deepak Ghimire, Kilho Lee, and Seong-heum Kim, “Loss-aware automatic selection of structured pruning criteria for deep neural network acceleration,” Image and Vision Computing, vol. 136, p. 104745, 2023.

Abstract: Structured pruning is a well-established technique for compressing neural networks, making them suitable for deployment in resource-limited edge devices. This study presents an efficient loss-aware automatic selection of structured pruning (LAASP) criteria for slimming and accelerating deep neural networks. The majority of pruning methods employ a sequential process consisting of three stages, 1) training, 2) pruning, and 3) fine-tuning, whereas the proposed pruning technique adopts a pruning-while-training approach that eliminates the first stage and integrates the second and third stages into a single cycle. The automatic selection of magnitude or similarity-based filter pruning criteria from a specified pool of criteria and the specific pruning layer at each pruning iteration is guided by the network's overall loss on a small subset of training data. To mitigate the abrupt accuracy drop due to pruning, the network is retrained briefly after each reduction of a predefined number of floating-point operations (FLOPs). The optimal pruning rates for each layer in the network are automatically determined, eliminating the need for manual allocation of fixed or variable pruning rates for each layer. Experiments on the VGGNet, ResNet, and MobileNet models on the CIFAR-10 and ImageNet benchmark datasets demonstrate the effectiveness of the proposed method. In particular, the ResNet56 and ResNet110 models on the CIFAR-10 dataset significantly improve the top-1 accuracy compared to state-of-the-art methods while reducing the network FLOPs by 52%. Furthermore, pruning the ResNet50 model on the ImageNet dataset reduces FLOPs by more than 42% with a negligible 0.33% drop in the top-5 accuracy.

@article{GHIMIRE2023104745,
    title = {Loss-aware automatic selection of structured pruning criteria for deep neural network acceleration},
    author = {Deepak Ghimire and Kilho Lee and Seong-heum Kim},
    journal = {Image and Vision Computing},
    volume = {136},
    pages = {104745},
    year = {2023},
    issn = {0262-8856},
    doi = {https://doi.org/10.1016/j.imavis.2023.104745},
    url = {https://www.sciencedirect.com/science/article/pii/S0262885623001191},
    keywords = {Deep neural networks, Structured pruning, Pruning criteria},
}

Note: Filter removal is based on VainF/Torch-Pruning.

alt text

Table of Contents

Requirements

  • Python 3.9.7
  • PyTorch 1.10.2
  • TorchVision 0.11.2
  • matplotlib 3.5.1
  • scipy 1.8.0

Models

Model Dataset Baseline Top@1 Acc. (%) Pruned Top@1 Acc. (%) Top@1 Acc. Drop (%) Pruned Top@5 Acc. (%) FLOPs RR (%)
Vgg16 CIFAR10 93.79 ± 0.23 93.90 ± 0.16 -0.11 - 34.6
Vgg16 CIFAR10 93.79 ± 0.23 93.79 ± 0.11 0.00 - 60.5
ResNet32 CIFAR10 93.12 ± 0.04 92.64 ± 0.09 0.48 - 53.3
ResNet56 CIFAR10 93.61 ± 0.11 93.04 ± 0.08 0.12 - 52.6
ResNet110 CIFAR10 94.41 ± 0.07 94.17 ± 0.16 0.24 - 52.5
ResNet110 CIFAR10 94.41 ± 0.07 93.58 ± 0.21 0.83 - 58.5
ResNet18 ImageNet 70.58 68.66 1.92 88.50 42.2
ResNet18 ImageNet 70.58 68.12 2.46 88.07 45.4
ResNet34 ImageNet 73.90 72.65 1.25 90.98 41.4
ResNet34 ImageNet 73.90 72.37 1.53 90.80 45.4
ResNet50 ImageNet 76.48 75.85 0.63 92.81 42.3
ResNet50 ImageNet 76.48 75.44 1.04 92.59 53.9
MobileNetV2 ImageNet 71.79 71.00 0.79 89.86 30.0
MobileNetV2 ImageNet 71.79 68.45 3.34 88.40 54.5

VGGNet on CIFAR-10

Training-Pruning

sh .\scripts\vgg16_cifar10\run_vgg16_pruning.sh

Evaluation

sh .\scripts\vgg16_cifar10\run_vgg16_eval.sh

ResNet on CIFAR-10

Training-Pruning

sh .\scripts\resnet_cifar10\run_resnet_cifar10_pruning.sh

Evaluation

sh .\scripts\resnet_cifar10\run_resnet_cifar10_eval.sh

ResNet on ImageNet

Prepare ImageNet dataset

  • Download the images from http://image-net.org/download-images

  • Extract the training data:

    mkdir train && mv ILSVRC2012_img_train.tar train/ && cd train
tar -xvf ILSVRC2012_img_train.tar && rm -f ILSVRC2012_img_train.tar
find . -name "*.tar" | while read NAME ; do mkdir -p "${NAME%.tar}"; tar -xvf "${NAME}" -C "${NAME%.tar}"; rm -f "${NAME}"; done
cd ..

  • Extract the validation data and move images to subfolders:

    mkdir val && mv ILSVRC2012_img_val.tar val/ && cd val && tar -xvf ILSVRC2012_img_val.tar
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash

Training-Pruning

sh .\scripts\resnet_imagenet\run_resnet_imagenet_pruning.sh

Evaluation

sh .\scripts\resnet_imagenet\run_resnet_imagenet_eval.sh