PPL QuantTool (PPQ) is a highly efficient neural network quantization tool with custimized IR, cuda based executor, automatic dispacher and powerful optimization passes. Together with OpenPPL ecosystem, we offer you this industrial-grade network deploy tool that empowers AI developers to unleash the full potential of AI hardware. With quantization and other optimizations, nerual network model can run 5~10x faster than ever.
PPL QuantTool 是一个工业级的神经网络量化工具：我们已经准备好了为你处理 maskrcnn 中复杂算子调度问题；esrgan 中的全网联合定点问题；或者是 transformer 中的大规模递归图融合，PPQ能够处理这些最复杂的网络量化任务，确保你的模型能够稳定部署在目标设备上。
PPQ 使用量化计算图(QIR)描述量化细节，即便在网络极度复杂的情况下，我们依然能够保证以正确的方法模拟硬件计算，从而降低模拟误差。我们知晓硬件的运算细节——在所有已知平台上，PPQ 的模拟误差不超过1%，且保证模拟误差不会指数级扩散。PPQ 有着自定义的量化算子库、网络执行器、调度器与异构执行能力，在网络量化与量化训练方面，使用 PPQ 比原生 PyTorch 快3 ~ 50倍。 借助 PPQ, OpenPPL, TensorRT, Tengine等框架，开发者可以将神经网络模型加速5 ~ 10倍，并部署到多种多样的目标终端，我们期待你将人工智慧真正带到千家万户之间。
PyTorch | TensorFlow | Caffe | ONNX | MMlab
TensorRT | OpenPPL-Cuda | OpenPPL-DSP | SNPE(Qualcomm) | NXP | Metax | Tengine(Developing) | ncnn | Openvino
| 使用例子(Examples) | 网络部署平台(Platform) | 输入模型格式(Format) | 链接(Link) | 相关视频(Video) |
|---|---|---|---|---|
新手上路 |
link | |||
| 量化你的第一个 pytorch 网络 | - | pytorch | link | |
| 量化你的第一个 onnx 网络 | - | onnx | link | |
| 量化你的第一个 caffe 网络 | - | caffe | link | |
| 走进 PPQ | - | onnx | link | link |
| 量化误差分析 | - | pytorch | link | link |
| 算子调度 | - | pytorch | link | link |
| 执行量化网络 | PPQ Executor | pytorch | link | |
| 启动 cuda kernel 加速执行 | PPQ Executor | pytorch | link | |
TensorRT |
||||
| 使用 Torch2trt 加速你的网络 | pytorch | pytorch | link | |
| TensorRT 量化训练 | TensorRT | pytorch | link | |
| TensorRT 后训练量化(PPQ) | TensorRT | onnx | link | |
| TensorRT fp32 部署 | TensorRT | onnx | link | |
| TensorRT 性能比较 | TensorRT | pytorch | link | |
onnxruntime |
||||
| 使用 onnxruntime 加速你的网络 | onnxruntime | onnx | link | link |
| onnx 后训练量化(PPQ) | onnxruntime | onnx | link | link |
| onnxruntime 性能比较 | onnxruntime | pytorch | link | link |
openvino |
||||
| 使用 openvino 加速你的网络 | openvino | onnx | ||
| openvino 后训练量化(PPQ) | openvino | onnx | ||
| openvino 性能比较 | openvino | pytorch | ||
snpe |
||||
| snpe 后训练量化(PPQ) | snpe | caffe | ||
ncnn |
||||
| ncnn 后训练量化(PPQ) | ncnn | onnx | link | |
OpenPPL |
||||
| ppl cuda 后训练量化(PPQ) | ppl cuda | onnx | link | |
自定义量化 |
||||
| 添加自定义量化平台 1 | - | pytorch | link | |
| 添加自定义量化平台 2 | - | pytorch | link |
Watch video tutorial series on www.bilibili.com, following are links of PPQ tutorial links(Only Chinese version).
- 安装教程: https://www.bilibili.com/video/BV1WS4y1N7Kn
- 使用教程: https://www.bilibili.com/video/BV1934y147p2
- 基础知识：https://www.bilibili.com/video/BV1gS4y1Y7KR
- 网络性能分析：https://www.bilibili.com/video/BV1oT4y1h73e
- 量化计算原理(Part 1)：https://www.bilibili.com/video/BV1fB4y1m7fJ
- 量化计算原理(Part 2)：https://www.bilibili.com/video/BV1qA4y1Q7Uh
- 图优化与量化模拟：https://www.bilibili.com/video/BV1Kr4y1n7cy
- 图调度与模式匹配：https://www.bilibili.com/video/BV1xY411A7ea
- 神经网络部署: https://www.bilibili.com/video/BV1t34y1E7Fz
- 其他教程: 等待缓慢更新...
To release the power of this advanced quantization tool, at least one CUDA computing device is required. Install CUDA from CUDA Toolkit, PPL Quantization Tool will use CUDA compiler to compile cuda kernels at runtime.
ATTENTION: For users of PyTorch, PyTorch might bring you a minimized CUDA libraries, which will not satisfy the requirement of this tool, you have to install CUDA from NVIDIA manually.
ATTENTION: Make sure your Python version is >= 3.6.0. PPL Quantization Tool is written with dialects that only supported by Python >= 3.6.0.
-
Install dependencies:
- For Linux User, use following command to install ninja:
sudo apt install ninja-build
- For Windows User:
- Download ninja.exe from https://github.com/ninja-build/ninja/releases, add it to Windows PATH Environment
- Download Visual Studio from https://visualstudio.microsoft.com, if you already got a c++ compiler, you can skip this step.
- Please Update Visual studio to 2019, otherwise there might be some compile error like: CxxFrameHandler4 unresolved, GSHandlerCheck unresolved.
- Add your c++ compiler to Windows PATH Environment, if you are using Visual Studio, it should be something like "C:\Program Files\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx86\x86"
-
Install PPQ from source:
- Run following code with your terminal(For windows user, use command line instead).
git clone https://github.com/openppl-public/ppq.git
cd ppq
pip install -r requirements.txt
python setup.py install- Wait for Python finish its installation and pray for bug free.
- Install PPQ from Pip:
- pre-built wheels are maintained in PPQ, you could simply install ppq with the following command(You should notice that install from pypi might get an outdated version ...)
python3 -m pip install ppq| WeChat Official Account | QQ Group |
|---|---|
| OpenPPL | 627853444 |
 |
 |
Email: openppl.ai@hotmail.com
We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion.
If you plan to contribute new features, utility functions, or extensions to the core, please first open an issue and discuss the feature with us. Sending a PR without discussion might end up resulting in a rejected PR because we might be taking the core in a different direction than you might be aware of.
PPQ is tested with models from mmlab-classification, mmlab-detection, mmlab-segamentation, mmlab-editing, here we listed part of out testing result.
- No quantization optimization procedure is applied with following models.
| Model | Type | Calibration | Dispatcher | Metric | PPQ(sim) | PPLCUDA | FP32 |
|---|---|---|---|---|---|---|---|
| Resnet-18 | Classification | 512 imgs | conservative | Acc-Top-1 | 69.50% | 69.42% | 69.88% |
| ResNeXt-101 | Classification | 512 imgs | conservative | Acc-Top-1 | 78.46% | 78.37% | 78.66% |
| SE-ResNet-50 | Classification | 512 imgs | conservative | Acc-Top-1 | 77.24% | 77.26% | 77.76% |
| ShuffleNetV2 | Classification | 512 imgs | conservative | Acc-Top-1 | 69.13% | 68.85% | 69.55% |
| MobileNetV2 | Classification | 512 imgs | conservative | Acc-Top-1 | 70.99% | 71.1% | 71.88% |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| retinanet | Detection | 32 imgs | pplnn | bbox_mAP | 36.1% | 36.1% | 36.4% |
| faster_rcnn | Detection | 32 imgs | pplnn | bbox_mAP | 36.6% | 36.7% | 37.0% |
| fsaf | Detection | 32 imgs | pplnn | bbox_mAP | 36.5% | 36.6% | 37.4% |
| mask_rcnn | Detection | 32 imgs | pplnn | bbox_mAP | 37.7% | 37.6% | 37.9% |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| deeplabv3 | Segamentation | 32 imgs | conservative | aAcc / mIoU | 96.13% / 78.81% | 96.14% / 78.89% | 96.17% / 79.12% |
| deeplabv3plus | Segamentation | 32 imgs | conservative | aAcc / mIoU | 96.27% / 79.39% | 96.26% / 79.29% | 96.29% / 79.60% |
| fcn | Segamentation | 32 imgs | conservative | aAcc / mIoU | 95.75% / 74.56% | 95.62% / 73.96% | 95.68% / 72.35% |
| pspnet | Segamentation | 32 imgs | conservative | aAcc / mIoU | 95.79% / 77.40% | 95.79% / 77.41% | 95.83% / 77.74% |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| srcnn | Editing | 32 imgs | conservative | PSNR / SSIM | 27.88% / 79.70% | 27.88% / 79.07% | 28.41% / 81.06% |
| esrgan | Editing | 32 imgs | conservative | PSNR / SSIM | 27.84% / 75.20% | 27.49% / 72.90% | 27.51% / 72.84% |
- PPQ(sim) stands for PPQ quantization simulator's result.
- Dispatcher stands for dispatching policy of PPQ.
- Classification models are evaluated with ImageNet, Detection and Segamentation models are evaluated with COCO dataset, Editing models are evaluated with DIV2K dataset.
- All calibration datasets are randomly picked from training data.
This project is distributed under the Apache License, Version 2.0.