FastDeploy: A C++ repository from HexToString

English | 简体中文

⚡️FastDeploy is an accessible and efficient deployment Development Toolkit. It covers 🔥critical AI models in the industry and provides 📦out-of-the-box deployment experience. It covers image classification, object detection, image segmentation, face detection, face recognition, human keypoint detection, OCR, semantic understanding and other tasks to meet developers' industrial deployment needs for multi-scenario, multi-hardware and multi-platform .

Potrait Segmentation	Image Matting	Semantic Segmentation	Real-Time Matting

OCR	Behavior Recognition	Object Detection	Pose Estimation

Face Alignment	3D Object Detection	Face Editing	Image Animation

📣 Recent Updates

🔥 2022.10.15：Release FastDeploy release v0.3.0
- New server-side deployment upgrade: support more CV model and NLP model
  - Integrate OpenVINO and provide a seamless deployment experience with other inference engines include TensorRT、ONNX Runtime、Paddle Inference；
  - Support one-click model quantization to improve model inference speed by 1.5 to 2 times on CPU & GPU platform. The supported quantized model are YOLOv7, YOLOv6, YOLOv5, etc.
  - New CV models include PP-OCRv3, PP-OCRv2, PP-TinyPose, PP-Matting, etc. and provides end-to-end deployment demos
  - New information extraction model is UIE, and provides end-to-end deployment demos.
🔥 2022.8.18：Release FastDeploy release v0.2.0
- New server-side deployment upgrade: faster inference performance, support more CV model
  - Release high-performance inference engine SDK based on x86 CPUs and NVIDIA GPUs, with significant increase in inference speed
  - Integrate Paddle Inference, ONNX Runtime, TensorRT and other inference engines and provide a seamless deployment experience
  - Supports full range of object detection models such as YOLOv7, YOLOv6, YOLOv5, PP-YOLOE and provides end-to-end deployment demos
  - Support over 40 key models and demo examples including face detection, face recognition, real-time portrait matting, image segmentation.
  - Support deployment in both Python and C++
- Supports Rockchip, Amlogic, NXP and other NPU chip deployment capabilities on edge device deployment
  - Release Lightweight Object Detection Picodet-NPU deployment demo, providing the full quantized inference capability for INT8.

Data Center and Cloud Deployment
Mobile and Edge Device Deployment
Community
Acknowledge
License

Data Center and Cloud Deployment

A Quick Start for Python SDK

Installation

Prerequisites

CUDA >= 11.2
cuDNN >= 8.0
python >= 3.6
OS: Linux x86_64/macOS/Windows 10

Install Fastdeploy SDK with CPU&GPU support

pip install fastdeploy-gpu-python -f https://www.paddlepaddle.org.cn/whl/fastdeploy.html

Conda Installation (Recommended)

conda config --add channels conda-forge && conda install cudatoolkit=11.2 cudnn=8.2

Install Fastdeploy SDK with only CPU support

pip install fastdeploy-python -f https://www.paddlepaddle.org.cn/whl/fastdeploy.html

Python Inference Example

Prepare models and pictures

wget https://bj.bcebos.com/paddlehub/fastdeploy/ppyoloe_crn_l_300e_coco.tgz
tar xvf ppyoloe_crn_l_300e_coco.tgz
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg

Test inference results

# For deployment of GPU/TensorRT, please refer to examples/vision/detection/paddledetection/python
import cv2
import fastdeploy.vision as vision


model = vision.detection.PPYOLOE("ppyoloe_crn_l_300e_coco/model.pdmodel",
                                 "ppyoloe_crn_l_300e_coco/model.pdiparams",
                                 "ppyoloe_crn_l_300e_coco/infer_cfg.yml")
im = cv2.imread("000000014439.jpg")
result = model.predict(im.copy())
print(result)

vis_im = vision.vis_detection(im, result, score_threshold=0.5)
cv2.imwrite("vis_image.jpg", vis_im)

A Quick Start for C++ SDK

Installation

Please refer to C++ Prebuilt Libraries Download

C++ Inference Example

Prepare models and pictures

wget https://bj.bcebos.com/paddlehub/fastdeploy/ppyoloe_crn_l_300e_coco.tgz
tar xvf ppyoloe_crn_l_300e_coco.tgz
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg

Test inference results

// For GPU/TensorRT deployment, please refer to examples/vision/detection/paddledetection/cpp
#include "fastdeploy/vision.h"

int main(int argc, char* argv[]) {
  namespace vision = fastdeploy::vision;
  auto model = vision::detection::PPYOLOE("ppyoloe_crn_l_300e_coco/model.pdmodel",
                                          "ppyoloe_crn_l_300e_coco/model.pdiparams",
                                          "ppyoloe_crn_l_300e_coco/infer_cfg.yml");
  auto im = cv::imread("000000014439.jpg");

  vision::DetectionResult res;
  model.Predict(&im, &res);

  auto vis_im = vision::Visualize::VisDetection(im, res, 0.5);
  cv::imwrite("vis_image.jpg", vis_im);
  return 0;
 }

For more deployment models, please refer to Vision Model Deployment Examples .

Supported Data Center and Cloud Model List🔥🔥🔥

Notes: ✅: already supported; ❔: to be supported in the future; ❌: not supported now;

Task	Model	API	Linux	Linux	Win	Win	Mac	Mac	Linux	Linux
---	---	---	X86 CPU	NVIDIA GPU	Intel CPU	NVIDIA GPU	Intel CPU	Arm CPU	AArch64 CPU	NVIDIA Jetson
Classification	PaddleClas/ResNet50	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/PP-LCNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/PP-LCNetv2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/EfficientNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/GhostNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/MobileNetV1	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/MobileNetV2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/MobileNetV3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/ShuffleNetV2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/SqueeezeNetV1.1	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/Inceptionv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/PP-HGNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Classification	PaddleClas/SwinTransformer	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	PaddleDetection/PP-YOLOE	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	PaddleDetection/PicoDet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	PaddleDetection/YOLOX	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	PaddleDetection/YOLOv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	PaddleDetection/PP-YOLO	Python/C++	✅	✅	✅	✅	✅	❌	❌	❌
Detection	PaddleDetection/PP-YOLOv2	Python/C++	✅	✅	✅	✅	✅	❌	❌	❌
Detection	PaddleDetection/FasterRCNN	Python/C++	✅	✅	✅	✅	✅	❌	❌	❌
Detection	Megvii-BaseDetection/YOLOX	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	WongKinYiu/YOLOv7	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	meituan/YOLOv6	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	ultralytics/YOLOv5	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	WongKinYiu/YOLOR	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	WongKinYiu/ScaledYOLOv4	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	ppogg/YOLOv5Lite	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	RangiLyu/NanoDetPlus	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
OCR	PaddleOCR/PP-OCRv2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
OCR	PaddleOCR/PP-OCRv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Segmentation	PaddleSeg/PP-LiteSeg	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Segmentation	PaddleSeg/PP-HumanSegLite	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Segmentation	PaddleSeg/HRNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Segmentation	PaddleSeg/PP-HumanSegServer	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Segmentation	PaddleSeg/Unet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Segmentation	PaddleSeg/Deeplabv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Detection	biubug6/RetinaFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Detection	Linzaer/UltraFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
FaceDetection	deepcam-cn/YOLOv5Face	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Detection	insightface/SCRFD	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Recognition	insightface/ArcFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Recognition	insightface/CosFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Recognition	insightface/PartialFC	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Face Recognition	insightface/VPL	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Matting	ZHKKKe/MODNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Matting	PaddleSeg/PP-Matting	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Matting	PaddleSeg/PP-HumanMatting	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Matting	PaddleSeg/ModNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Information Extraction	PaddleNLP/UIE	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅

Edge-Side Deployment

EasyEdge Edge-Side Deployment

ARM Linux System

EasyEdge Deployment on Mobile Devices

EasyEdge Customized Deployment

Replace Model With Another One

Paddle Lite NPU Deployment

Rexchip-NPU / Amlogic-NPU / NXP-NPU

Supported Edge-Side Model List

	Model	Size (MB)	Linux	Android	iOS	Linux	Linux	Linux	TBD...
---	---	---	ARM CPU	ARM CPU	ARM CPU	Rockchip-NPU RV1109 RV1126 RK1808	Amlogic-NPU A311D S905D C308X	NXPNPU i.MX 8M Plus	TBD...｜
Classification	PP-LCNet	11.9	✅	✅	✅	--	--	--	--
Classification	PP-LCNetv2	26.6	✅	✅	✅	--	--	--	--
Classification	EfficientNet	31.4	✅	✅	✅	--	--	--	--
Classification	GhostNet	20.8	✅	✅	✅	--	--	--	--
Classification	MobileNetV1	17	✅	✅	✅	--	--	--	--
Classification	MobileNetV2	14.2	✅	✅	✅	--	--	--	--
Classification	MobileNetV3	22	✅	✅	✅	❔	❔	❔	❔
Classification	ShuffleNetV2	9.2	✅	✅	✅	--	--	--	--
Classification	SqueezeNetV1.1	5	✅	✅	✅
Classification	Inceptionv3	95.5	✅	✅	✅	--	--	--	--
Classification	PP-HGNet	59	✅	✅	✅	--	--	--	--
Classification	SwinTransformer_224_win7	352.7	✅	✅	✅	--	--	--	--
Detection	PP-PicoDet_s_320_coco	4.1	✅	✅	✅	--	--	--	--
Detection	PP-PicoDet_s_320_lcnet	4.9	✅	✅	✅	✅	✅	✅	❔
Detection	CenterNet	4.8	✅	✅	✅	--	--	--	--
Detection	YOLOv3_MobileNetV3	94.6	✅	✅	✅	--	--	--	--
Detection	PP-YOLO_tiny_650e_coco	4.4	✅	✅	✅	--	--	--	--
Detection	SSD_MobileNetV1_300_120e_voc	23.3	✅	✅	✅	--	--	--	--
Detection	PP-YOLO_ResNet50vd	188.5	✅	✅	✅	--	--	--	--
Detection	PP-YOLOv2_ResNet50vd	218.7	✅	✅	✅	--	--	--	--
Detection	PP-YOLO_crn_l_300e_coco	209.1	✅	✅	✅	--	--	--	--
Detection	YOLOv5s	29.3	✅	✅	✅	--	--	--	--
Face Detection	BlazeFace	1.5	✅	✅	✅	--	--	--	--
Face Detection	RetinaFace	1.7	✅	❌	❌	--	--	--	--
Keypoint Detection	PP-TinyPose	5.5	✅	✅	✅	❔	❔	❔	❔
Segmentation	PP-LiteSeg(STDC1)	32.2	✅	✅	✅	--	--	--	--
Segmentation	PP-HumanSeg-Lite	0.556	✅	✅	✅	--	--	--	--
Segmentation	HRNet-w18	38.7	✅	✅	✅	--	--	--	--
Segmentation	PP-HumanSeg-Server	107.2	✅	✅	✅	--	--	--	--
Segmentation	Unet	53.7	❌	✅	❌	--	--	--	--
OCR	PP-OCRv1	2.3+4.4	✅	✅	✅	--	--	--	--
OCR	PP-OCRv2	2.3+4.4	✅	✅	✅	--	--	--	--
OCR	PP-OCRv3	2.4+10.6	✅	✅	✅	❔	❔	❔	❔
OCR	PP-OCRv3-tiny	2.4+10.7	✅	✅	✅	--	--	--	--

Community

If you have any question or suggestion, please give us your valuable input via GitHub Issues
Join Us👬：
- Slack：Join our Slack community and chat with other community members about ideas
- WeChat：join our WeChat community and chat with other community members about ideas

Acknowledge

We sincerely appreciate the open-sourced capabilities in EasyEdge as we adopt it for the SDK generation and download in this project.

License

FastDeploy is provided under the Apache-2.0.

HexToString/FastDeploy

📣 Recent Updates

Contents

Data Center and Cloud Deployment

A Quick Start for Python SDK

Installation

Prerequisites

Install Fastdeploy SDK with CPU&GPU support

Conda Installation (Recommended)

Install Fastdeploy SDK with only CPU support

Python Inference Example

A Quick Start for C++ SDK

Installation

C++ Inference Example

Supported Data Center and Cloud Model List🔥🔥🔥

Edge-Side Deployment

EasyEdge Edge-Side Deployment

EasyEdge Deployment on Mobile Devices

EasyEdge Customized Deployment

Paddle Lite NPU Deployment

Supported Edge-Side Model List

Community

Acknowledge

License