Reduced storage direct tensor ring decomposition for convolutional neural networks compression

Data and baseline models

In experiments CIFAR-10 and ImageNet datasets are used. For CIFAR-10 datasets the pretrained models are downloaded from torchvision library. Models for CIFAR-10 dataset are trained from scratch.

Requirements

pip install -r requirements
Download the ImageNet dataset from http://www.image-net.org/

Training baseline models

After cloning the repository you can train baseline CIFAR-10 models by running the following command.

python cifar.py -net <net name> -mode <mode> -lr <lr> -epochs <epochs> -wd <wd> -b <b> -momentum <momentum>

where:

<net name> - name of neural network, one from the following list:
- cifar10_resnet20
- cifar10_resnet32
- cifar10_resnet56
- cifar10_vggnet
<mode> - there are two available modes, train for training and fine_tune for fine tuning
<lr> - learning rate
<epochs> - number of epochs
wd - weight decay
<b> - batch size
<momentum> - value of momentum

The following command shows the example of training ResNet-20 network from scratch.

python cifar.py -net cifar10_resnet20 -mode train -lr 0.1 -epochs 200 -wd 1e-4 -b 128 -momentum 0.9

Compressing baseline models

For compression of baseline models use the following command:

python decompose_network.py -net <net name> -weights <weight path> -p <p>

where,

<net name> - name of neural network, one from the following list:
- cifar10_resnet20
- cifar10_resnet32
- cifar10_resnet56
- cifar10_vggnet
- imagenet_resnet18
- imagenet_resnet34
<weights> - path of baseline network weights (only for CIFAR-10 models)
<p>- prescribed relative error of tensor ring decomposition (float number in range [0,1])

The following command shows the example of compressing ResNet-18 network.

python decompose_network.py -net imagenet_resnet18 -p 0.5

Fine-tuning compressed network

For fine-tuning compressed network for CIFAR-10 dataset, use the following command:

python cifar.py -net <net name> -weights <weight path> -mode <mode> -lr <lr> -epochs <epochs> -wd <wd> -b <b> -momentum <momentum>

<weight path> is the path of compressed weights and <mode> has to be changed to fine_tune. The concrete example of fine tuning compressed ResNet-20 is shown below.

python cifar.py -net cifar10_resnet20 -weights decomposed_weights/tr_cifar10_resnet20_0_5.pth -mode fine_tune -lr 0.01 -epochs 160 -wd <wd> -b 128 -momentum 0

For fine-tuning compressed networks for ImageNet dataset used the following command.

python imagenet.py -weights <weight path> -lr <lr> -epochs <epochs> -wd <wd> -b <b> -momentum <momentum> -train <train path> -val <val path> -workers <workers>

where

<train path> - path of ImageNet training dataset
<val path> - path of ImageNet validation dataset
<workers> - number of workers

Example of fine tuning compressed ResNet-18 network is shown below.