/ppq

PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.

Primary LanguagePythonApache License 2.0Apache-2.0

PPL Quantization Tool 0.6.5(PPL 量化工具)

PPL QuantTool (PPQ) is a highly efficient neural network quantization tool with custimized IR, cuda based executor, automatic dispacher and powerful optimization passes. Together with OpenPPL ecosystem, we offer you this industrial-grade network deploy tool that empowers AI developers to unleash the full potential of AI hardware. With quantization and other optimizations, nerual network model can run 5~10x faster than ever.

PPL QuantTool 是一个高效的工业级神经网络量化工具。 PPQ 被设计为一个灵活而全面的神经网络离线量化工具，我们允许你控制对量化进行细致入微的控制，同时严格控制硬件模拟误差。即便在网络极度复杂的情况下，我们依然能够得到正确的网络量化结果。PPQ 有着自定义的量化算子库、网络执行器、神经网络调度器、量化计算图等独特设计，这些为量化而设计的组件相互协作，共同构成了这一先进神经网络量化框架。借助 PPQ, OpenPPL, TensorRT, Tengine，ncnn等框架，你可以将神经网络模型提速 10 ~ 100 倍，并部署到多种多样的目标硬件平台，我们期待你将人工智慧带到千家万户之间。

目前 PPQ 使用 onnx(opset 11 ~ 13) 模型文件作为输入，覆盖常用的 90 余种 onnx 算子类型。如果你是 Pytorch, tensorflow 的用户，你可以使用这些框架提供的方法将模型转换到 onnx。PPQ 支持向 TensorRT, OpenPPL, Openvino, ncnn, Onnxruntime, Tengine, Snpe 等多个推理引擎生成目标文件并完成部署。借助算子自定义与平台自定义功能，你还可以将 PPQ 的量化能力扩展到其他可能的硬件上。

Code Example

使用例子(Examples) 网络部署平台(Platform) 输入模型格式(Format) 链接(Link) 相关视频(Video)
新手上路 link
量化你的第一个 pytorch 网络 - pytorch link
量化你的第一个 onnx 网络 - onnx link
量化你的第一个 caffe 网络 - caffe link
走进 PPQ - onnx link link
量化函数 - - link
量化参数选择 - - link
量化误差分析 - pytorch link link
算子调度 - pytorch link link
执行量化网络 PPQ Executor pytorch link
启动 cuda kernel 加速执行 PPQ Executor pytorch link
TensorRT
使用 Torch2trt 加速你的网络 pytorch pytorch link link
TensorRT 量化训练 TensorRT pytorch link link
TensorRT 后训练量化(PPQ) TensorRT onnx link link
TensorRT fp32 部署 TensorRT onnx link link
TensorRT 性能比较 TensorRT pytorch link link
TensorRT 性能分析工具 TensorRT pytorch link link
onnxruntime
使用 onnxruntime 加速你的网络 onnxruntime onnx link link
onnx 后训练量化(PPQ) onnxruntime onnx link link
onnxruntime 性能比较 onnxruntime pytorch link link
openvino
使用 openvino 加速你的网络 openvino onnx link
openvino 量化训练 openvino pytorch link
openvino 后训练量化(PPQ) openvino onnx link
openvino 性能比较 openvino pytorch link
snpe
snpe 后训练量化(PPQ) snpe caffe link
ncnn
ncnn 后训练量化(PPQ) ncnn onnx link
OpenPPL
ppl cuda 后训练量化(PPQ) ppl cuda onnx link
自定义量化
添加自定义量化平台 1 - pytorch link
添加自定义量化平台 2 - pytorch link
注册量化代理函数 - pytorch link
自定义量化算子 - pytorch link
绕过与量化无关的算子 - pytorch link
其他
onnx 格式转换 - onnx link

Video Tutorial(Bilibili 视频教程)

Watch video tutorial series on www.bilibili.com, following are links of PPQ tutorial links(Only Chinese version).

Installation

To release the power of this advanced quantization tool, at least one CUDA computing device is required. Install CUDA from CUDA Toolkit, PPL Quantization Tool will use CUDA compiler to compile cuda kernels at runtime.

ATTENTION: For users of PyTorch, PyTorch might bring you a minimized CUDA libraries, which will not satisfy the requirement of this tool, you have to install CUDA from NVIDIA manually.

ATTENTION: Make sure your Python version is >= 3.6.0. PPL Quantization Tool is written with dialects that only supported by Python >= 3.6.0.

  • Install dependencies:

    • For Linux User, use following command to install ninja:
    sudo apt install ninja-build
    • For Windows User:
      • Download ninja.exe from https://github.com/ninja-build/ninja/releases, add it to Windows PATH Environment
      • Download Visual Studio 2019 from https://visualstudio.microsoft.com, if you already got a c++ compiler, you can skip this step.
      • Add your c++ compiler to Windows PATH Environment, if you are using Visual Studio, it should be something like "C:\Program Files\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx86\x86"
      • Update pytorch to 1.10+.

  • Install PPQ from source:

  1. Run following code with your terminal(For windows user, use command line instead).
git clone https://github.com/openppl-public/ppq.git
cd ppq
pip install -r requirements.txt
python setup.py install
  1. Wait for Python finish its installation and pray for bug free.
  • Install PPQ from Pip:
  1. pre-built wheels are maintained in PPQ, you could simply install ppq with the following command(You should notice that install from pypi might get an outdated version ...)
python3 -m pip install ppq

Contact Us

WeChat Official Account QQ Group
OpenPPL 627853444
OpenPPL QQGroup

Email: openppl.ai@hotmail.com

Other Resources

Contributions

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion.

If you plan to contribute new features, utility functions, or extensions to the core, please first open an issue and discuss the feature with us. Sending a PR without discussion might end up resulting in a rejected PR because we might be taking the core in a different direction than you might be aware of.

Benchmark

PPQ is tested with models from mmlab-classification, mmlab-detection, mmlab-segamentation, mmlab-editing, here we listed part of out testing result.

  • No quantization optimization procedure is applied with following models.
Model Type Calibration Dispatcher Metric PPQ(sim) PPLCUDA FP32
Resnet-18 Classification 512 imgs conservative Acc-Top-1 69.50% 69.42% 69.88%
ResNeXt-101 Classification 512 imgs conservative Acc-Top-1 78.46% 78.37% 78.66%
SE-ResNet-50 Classification 512 imgs conservative Acc-Top-1 77.24% 77.26% 77.76%
ShuffleNetV2 Classification 512 imgs conservative Acc-Top-1 69.13% 68.85% 69.55%
MobileNetV2 Classification 512 imgs conservative Acc-Top-1 70.99% 71.1% 71.88%
---- ---- ---- ---- ---- ---- ---- ----
retinanet Detection 32 imgs pplnn bbox_mAP 36.1% 36.1% 36.4%
faster_rcnn Detection 32 imgs pplnn bbox_mAP 36.6% 36.7% 37.0%
fsaf Detection 32 imgs pplnn bbox_mAP 36.5% 36.6% 37.4%
mask_rcnn Detection 32 imgs pplnn bbox_mAP 37.7% 37.6% 37.9%
---- ---- ---- ---- ---- ---- ---- ----
deeplabv3 Segamentation 32 imgs conservative aAcc / mIoU 96.13% / 78.81% 96.14% / 78.89% 96.17% / 79.12%
deeplabv3plus Segamentation 32 imgs conservative aAcc / mIoU 96.27% / 79.39% 96.26% / 79.29% 96.29% / 79.60%
fcn Segamentation 32 imgs conservative aAcc / mIoU 95.75% / 74.56% 95.62% / 73.96% 95.68% / 72.35%
pspnet Segamentation 32 imgs conservative aAcc / mIoU 95.79% / 77.40% 95.79% / 77.41% 95.83% / 77.74%
---- ---- ---- ---- ---- ---- ---- ----
srcnn Editing 32 imgs conservative PSNR / SSIM 27.88% / 79.70% 27.88% / 79.07% 28.41% / 81.06%
esrgan Editing 32 imgs conservative PSNR / SSIM 27.84% / 75.20% 27.49% / 72.90% 27.51% / 72.84%
  • PPQ(sim) stands for PPQ quantization simulator's result.
  • Dispatcher stands for dispatching policy of PPQ.
  • Classification models are evaluated with ImageNet, Detection and Segamentation models are evaluated with COCO dataset, Editing models are evaluated with DIV2K dataset.
  • All calibration datasets are randomly picked from training data.

License

This project is distributed under the Apache License, Version 2.0.