/Synflow_SNIP_GraSP

Comparison of method "Pruning at initialization prior to training" (Synflow/SNIP/GraSP) in PyTorch

Primary LanguagePython

Comparison of Synflow/SNIP/GraSP

Implementation of Synaptic flow, Single-shot Network Pruning, Gradient Signal Preservation in pytorch v1.6.
Aims to compare pruning method.

  1. Synaptic Flow: Pruning Neural Networks Without Any Data By Iteratively Conserving Synaptic Flow
    Paper: https://arxiv.org/pdf/2006.05467.pdf
    Official repo: https://github.com/ganguli-lab/Synaptic-Flow

  2. SNIP: Single-shot Network Pruning based on Connection Sensitivity(ICLR2019)
    Paper: https://github.com/namhoonlee/snip-public
    Official repo: https://github.com/namhoonlee/snip-public

  3. GraSP: Picking Winning Tickets Before Training By Preserving Gradient Flow(ICLR2020)
    Paper: https://openreview.net/pdf?id=SkgsACVKPH
    Official repo: https://github.com/alecwangcq/GraSP

Onging process

Follwing pruning method(Pruning at initialization prior to training or while training) will be implemented.

  • Synaptic flow
  • GraSP
  • SNIP
  • Random
  • Magnitude
  • Plot

Run

❯ python run.py --help
usage: run.py [-h] [--SEED SEED] [--gpu GPU] [--model MODEL]
              [--dataset DATASET] [--batch_size BATCH_SIZE]
              [--method_list METHOD_LIST [METHOD_LIST ...]]
              [--ratio_list RATIO_LIST [RATIO_LIST ...]]

Pruning tester.

optional arguments:
  -h, --help            show this help message and exit
  --SEED SEED           Seed number
  --gpu GPU             GPU id to use
  --model MODEL         Model to test, torchvision model name
  --dataset DATASET     Dataset in torchvision.datasets ex) CIFAR10, CIFAR100,
                        MNIST
  --batch_size BATCH_SIZE
                        Batch size, default: 64
  --method_list METHOD_LIST [METHOD_LIST ...]
                        Pruning method(Rand/Mag/Synflow) list run
                        sequentially. ex) --method_list Synflow Rand Mag
  --ratio_list RATIO_LIST [RATIO_LIST ...]
                        List of pruning ratio. ex) --ratio_list 0 0.5 0.9 0.95
                        0.99

Example

python run.py --model resnet18 --dataset MNIST --method_list Rand Mag Synflow --ratio_list 0 0.5 0.9 0.99