This repository implements various quantization methods listed below for beginners of the quantization of deep neural networks.
For the first, to train a pre-trained and full-precision model,
python3 main.py --name FP --gpu 0
Then you can load the pre-trained checkpoint to initialize at low-precision training,
python3 main.py --name 4B --gpu 0 --quatizer lsq --bitw 4 --bita 4 --init pre-trained/full-precision/model/ckpt.pth
Evaludation of ResNet-18 on CIFAR-100 (full-precision: 77.05)
| Bit-W / Bit-A | 2 / 2 | 3 / 3 | 4 / 4 | 5 / 5 | 8 / 8 |
|---|---|---|---|---|---|
| LSQ [4] | 00.00 | 00.00 | 76.94 | 00.00 | 00.00 |
| SAT [5] | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
| QIL [3] | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
| PACT [2] | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
| DoReFa-net [1] | 00.00 | 00.00 | 00.00 | 00.00 | 00.00 |
[1] Zhou, Shuchang, et al. "Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients." arXiv preprint arXiv:1606.06160 (2016).
[2] Choi, Jungwook, et al. "Pact: Parameterized clipping activation for quantized neural networks." arXiv preprint arXiv:1805.06085 (2018).
[3] Jung, Sangil, et al. "Learning to quantize deep networks by optimizing quantization intervals with task loss." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.
[4] Esser, Steven K., et al. "Learned step size quantization." arXiv preprint arXiv:1902.08153 (2019).
[5] Jin, Qing, Linjie Yang, and Zhenyu Liao. "Towards efficient training for neural network quantization." arXiv preprint arXiv:1912.10207 (2019).