/AdderNet

Code for paper " AdderNet: Do We Really Need Multiplications in Deep Learning?"

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

AdderNet: Do We Really Need Multiplications in Deep Learning?

This code is a demo of CVPR 2020 paper AdderNet: Do We Really Need Multiplications in Deep Learning?

We present adder networks (AdderNets) to trade massive multiplications in deep neural networks, especially convolutional neural networks (CNNs), for much cheaper additions to reduce computation costs. In AdderNets, we take the L1-norm distance between filters and input feature as the output response. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset without any multiplication in convolution layer.

Run python main.py to train on CIFAR-10.

Classification results on CIFAR-10 and CIFAR-100 datasets.

Model Method CIFAR-10 CIFAR-100
VGG-small ANN 93.72% 72.64%
PKKD ANN 95.03% 76.94%
SLAC ANN 93.96% 73.63%
ResNet-20 ANN 92.02% 67.60%
PKKD ANN 92.96% 69.93%
SLAC ANN 92.29% 68.31%
ShiftAddNet* 89.32%(160epoch) -
ResNet-32 ANN 93.01% 69.17%
PKKD ANN 93.62% 72.41%
SLAC ANN 93.24% 69.83%

Classification results on ImageNet dataset.

Model Method Top-1 Acc Top-5 Acc
ResNet-18 CNN 69.8% 89.1%
ANN 67.0% 87.6%
PKKD ANN 68.8% 88.6%
SLAC ANN 67.7% 87.9%
ResNet-50 CNN 76.2% 92.9%
ANN 74.9% 91.7%
PKKD ANN 76.8% 93.3%
SLAC ANN 75.3% 92.6%

*ShiftAddNet used different training setting.

Super-Resolution results on several SR datasets.

Scale Model Method Set5 (PSNR/SSIM) Set14 (PSNR/SSIM) B100 (PSNR/SSIM) Urban100 (PSNR/SSIM)
×2 VDSR CNN 37.53/0.9587 33.03/0.9124 31.90/0.8960 30.76/0.9140
ANN 37.37/0.9575 32.91/0.9112 31.82/0.8947 30.48/0.9099
EDSR CNN 38.11/0.9601 33.92/0.9195 32.32/0.9013 32.93/0.9351
ANN 37.92/0.9589 33.82/0.9183 32.23/0.9000 32.63/0.9309
×3 VDSR CNN 33.66/0.9213 29.77/0.8314 28.82/0.7976 27.14/0.8279
ANN 33.47/0.9151 29.62/0.8276 28.72/0.7953 26.95/0.8189
EDSR CNN 34.65/0.9282 30.52/0.8462 29.25/0.8093 28.80/0.8653
ANN 34.35/0.9212 30.33/0.8420 29.13/0.8068 28.54/0.8555
×4 VDSR CNN 31.35/0.8838 28.01/0.7674 27.29/0.7251 25.18/0.7524
ANN 31.27/0.8762 27.93/0.7630 27.25/0.7229 25.09/0.7445
EDSR CNN 32.46/0.8968 28.80/0.7876 27.71/0.7420 26.64/0.8033
ANN 32.13/0.8864 28.57/0.7800 27.58/0.7368 26.33/0.7874

Adversarial robustness on CIFAR-10 under white-box attacks without adversarial training.

Model Method Clean FGSM BIM7 PGD7 MIM5 RFGSM5
ResNet-20 CNN 92.68 16.33 0.00 0.00 0.01 0.00
ANN 91.72 18.42 0.00 0.00 0.04 0.00
CNN-R 90.62 17.23 3.46 3.67 4.23 0.06
ANN-R 90.95 29.93 29.30 29.72 32.25 3.38
ANN-R-AWN 90.55 45.93 42.62 43.39 46.52 18.36
ResNet-32 CNN 92.78 23.55 0.00 0.01 0.10 0.00
ANN 92.48 35.85 0.03 0.11 1.04 0.02
CNN-R 91.32 20.41 5.15 5.27 6.09 0.07
ANN-R 91.68 19.74 15.96 16.08 17.48 0.07
ANN-R-AWN 91.25 61.30 59.41 59.74 61.54 39.79

Comparisons of mAP on PASCAL VOC.

Model Backbone Neck mAP
Faster R-CNN Conv R50 Conv 79.5
FCOS Conv R50 Conv 79.1
RetinaNet Conv R50 Conv 77.3
FoveaBox Conv R50 Conv 76.6
Adder-FCOS Adder R50 Adder 76.5

Requirements

  • python 3
  • pytorch >= 1.1.0
  • torchvision

Preparation

You can follow pytorch/examples to prepare the ImageNet data.

The pretrained models are available in google drive or baidu cloud (access code:126b)

Usage

Run python main.py to train on CIFAR-10.

Run python test.py --data_dir 'path/to/imagenet_root/' to evaluate on ImageNet val set. You will achieve 74.9% Top accuracy and 91.7% Top-5 accuracy on the ImageNet dataset using ResNet-50.

Run python test.py --dataset cifar10 --model_dir models/ResNet20-AdderNet.pth --data_dir 'path/to/cifar10_root/' to evaluate on CIFAR-10. You will achieve 91.8% accuracy on the CIFAR-10 dataset using ResNet-20.

The inference and training of AdderNets is slow since the adder filters is implemented without cuda acceleration. You can write cuda to achieve higher inference speed.

Citation

@article{AdderNet,
	title={AdderNet: Do We Really Need Multiplications in Deep Learning?},
	author={Chen, Hanting and Wang, Yunhe and Xu, Chunjing and Shi, Boxin and Xu, Chao and Tian, Qi and Xu, Chang},
	journal={CVPR},
	year={2020}
}

Contributing

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion.

If you plan to contribute new features, utility functions or extensions to the core, please first open an issue and discuss the feature with us. Sending a PR without discussion might end up resulting in a rejected PR, because we might be taking the core in a different direction than you might be aware of.