Intel® Neural Compressor, formerly known as Intel® Low Precision Optimization Tool, an open-source Python library running on Intel CPUs and GPUs, which delivers unified interfaces across multiple deep learning frameworks for popular network compression technologies, such as quantization, pruning, knowledge distillation. This tool supports automatic accuracy-driven tuning strategies to help user quickly find out the best quantized model. It also implements different weight pruning algorithms to generate pruned model with predefined sparsity goal and supports knowledge distillation to distill the knowledge from the teacher model to the student model. Intel® Neural Compressor has been one of the critical AI software components in Intel® oneAPI AI Analytics Toolkit.
Note: GPU support is under development.
Visit the Intel® Neural Compressor online document website at: https://intel.github.io/neural-compressor.
Prerequisites
- Python version: 3.7 or 3.8 or 3.9 or 3.10
Install on Linux
# install stable basic version from pip
pip install neural-compressor
# install stable full version from pip (including GUI)
pip install neural-compressor-full
# install nightly basic version from pip
pip install -i https://test.pypi.org/simple/ neural-compressor
# install nightly full version from pip (including GUI)
pip install -i https://test.pypi.org/simple/ neural-compressor-full
# install stable basic version from from conda
conda install neural-compressor -c conda-forge -c intel
# install stable full version from from conda (including GUI)
conda install neural-compressor-full -c conda-forge -c intel
More installation methods can be found at Installation Guide. Please check out our FAQ for more details.
- Quantization with Python API
# A TensorFlow Example
pip install tensorflow
# Prepare fp32 model
wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_6/mobilenet_v1_1.0_224_frozen.pb
import tensorflow as tf
from neural_compressor.experimental import Quantization, common
tf.compat.v1.disable_eager_execution()
quantizer = Quantization()
quantizer.model = './mobilenet_v1_1.0_224_frozen.pb'
dataset = quantizer.dataset('dummy', shape=(1, 224, 224, 3))
quantizer.calib_dataloader = common.DataLoader(dataset)
quantizer.fit()
- Quantization with GUI
# An ONNX Example
pip install onnx==1.9.0 onnxruntime==1.10.0 onnxruntime-extensions
# Prepare fp32 model
wget https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx
# Start GUI
inc_bench
![Architecture](https://raw.githubusercontent.com/KimBioInfoStudio/neural-compressor/master/./docs/imgs/INC_GUI.gif)
- Quantization with Auto-coding API (Experimental)
from neural_coder import auto_quant
auto_quant(
code="https://github.com/huggingface/transformers/blob/v4.21-release/examples/pytorch/text-classification/run_glue.py",
args="--model_name_or_path albert-base-v2 \
--task_name sst2 \
--do_eval \
--output_dir result \
--overwrite_output_dir",
)
Intel® Neural Compressor supports systems based on Intel 64 architecture or compatible processors, specially optimized for the following CPUs:
- Intel Xeon Scalable processor (formerly Skylake, Cascade Lake, Cooper Lake, and Icelake)
- Future Intel Xeon Scalable processor (code name Sapphire Rapids)
- OS version: CentOS 8.4, Ubuntu 20.04
- Python version: 3.7, 3.8, 3.9, 3.10
Framework | TensorFlow | Intel TensorFlow | PyTorch | IPEX | ONNX Runtime | MXNet |
---|---|---|---|---|---|---|
Version | 2.9.1 2.8.2 2.7.3 | 2.9.1 2.8.0 2.7.0 | 1.12.0+cpu 1.11.0+cpu 1.10.0+cpu |
1.12.0 1.11.0 1.10.0 |
1.11.0 1.10.0 1.9.0 |
1.8.0 1.7.0 1.6.0 |
Note: Please set the environment variable TF_ENABLE_ONEDNN_OPTS=1 to enable oneDNN optimizations if you are using TensorFlow from v2.6 to v2.8. oneDNN has been fully default from TensorFlow v2.9.
Intel® Neural Compressor validated 420+ examples for quantization with performance speedup geomean 2.2x and up to 4.2x on VNNI while minimizing the accuracy loss. And also provided 30+ pruning and knowledge distillation samples.
More details for validated models are available here.
![Architecture](https://raw.githubusercontent.com/KimBioInfoStudio/neural-compressor/master/./docs/imgs/release_data.png)
- 地址标准化服务AI深度学习模型推理优化实践 (Aug 2022)
- Accelerate AI Inference without Sacrificing Accuracy
- Accelerate Deep Learning with Intel® Extension for TensorFlow*
- PyTorch Inference Acceleration with Intel® Neural Compressor (Jun 2022)
- Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration (Jun 2022)
- Intel® Deep Learning Boost - Boost Network Security AI Inference Performance in Google Cloud Platform (GCP) (Apr 2022)
- Intel® Neural Compressor joined PyTorch ecosystem tool (Apr 2022)
- Dynamic Quantization with Intel Neural Compressor and Transformers (Mar 2022)
Please check out our full publication list.
- Release Information
- Contribution Guidelines
- Legal Information
- Security Policy
- Intel® Neural Compressor Website
We are actively hiring. Please send your resume to inc.maintainers@intel.com if you have interests in model compression techniques.