/play_with_tensorrt

Sample projects for TensorRT in C++

Primary LanguageC++Apache License 2.0Apache-2.0

Play with TensorRT

  • Sample projects to use TensorRT in C++ for multi-platform
  • Typical project structure is like the following diagram
    • 00_doc/design.jpg

Target

  • Platform
    • Linux (x64)
    • Linux (aarch64)
    • Windows (x64). Visual Studio 2019

Usage

./main [input]

 - input = blank
    - use the default image file set in source code (main.cpp)
    - e.g. ./main
 - input = *.mp4, *.avi, *.webm
    - use video file
    - e.g. ./main test.mp4
 - input = *.jpg, *.png, *.bmp
    - use image file
    - e.g. ./main test.jpg
 - input = number (e.g. 0, 1, 2, ...)
    - use camera
    - e.g. ./main 0
- input = jetson
    - use camera via gstreamer on Jetson
    - e.g. ./main jetson

How to build a project

0. Requirements

  • OpenCV 4.x
  • CUDA + cuDNN
  • TensorRT 8.x
    • In case you have build errors related to TensorRT location, modify cmake settings for it in InferenceHelper/inference_helper/CMakeLists.txt

1. Download

  • Download source code and pre-built libraries 
    git clone https://github.com/iwatake2222/play_with_tensorrt.git
cd play_with_tensorrt
git submodule update --init
sh InferenceHelper/third_party/download_prebuilt_libraries.sh
  • Download models 
    sh ./download_resource.sh

2-a. Build in Linux

  • Build and run 
    cd pj_tensorrt_cls_mobilenet_v2   # for example
mkdir -p build && cd build
cmake ..
make
./main

2-b. Build in Windows (Visual Studio)

  • Configure and Generate a new project using cmake-gui for Visual Studio 2019 64-bit
    • Where is the source code : path-to-play_with_tensorrt/pj_tensorrt_cls_mobilenet_v2 (for example)
    • Where to build the binaries : path-to-build (any)
  • Open main.sln
  • Set main project as a startup project, then build and run!

Configuration for TensorRT

You don't need to change any configuration for TensorRT, but you can change it if you want.

Model format

  • The model file name is specified in xxx_engine.cpp . Please find MODEL_NAME definition
  • inference_helper_tensorrt.cpp automatically converts model according to the model format (extension)
    • .onnx : convert the model from onnx to trt, and save the converted trt model
    • .uff : convert the model from uff to trt, and save the converted trt model (WIP)
    • .trt : use pre-converted trt model
  • If *.trt file exists, InferenceHelper will use it to avoid re-conversion to save time
    • If you want to re-convert (for example, when you try another conversion settings), please delete resource/model/*.trt
    • Also, if you want to re-convert with INT8 calibration, please delete CalibrationTable_cal.txt

DLA Cores (NVDLA)

  • GPU is used by default
  • Call SetDlaCore(0) or SetDlaCore(1) to use DLA

Model conversion settings

  • The parameters for model conversion is defiend in inference_helper_tensorrt.cpp
  • USE_FP16
    • define this for FP16 inference
  • USE_INT8_WITHOUT_CALIBRATION
    • define this for INT8 inference without calibration (I can't get good result with this)
  • USE_INT8_WITH_CALIBRATION
    • define this for INT8 inference (you also need int8 calibration)
  • OPT_MAX_WORK_SPACE_SIZE
    • 1 << 30
  • OPT_AVG_TIMING_ITERATIONS
    • not in use
  • OPT_MIN_TIMING_ITERATIONS
    • not in use
  • Parameters for Quantization Calibration
    • CAL_DIR
      • directory containing calibration images (ppm in the same size as model input size)
    • CAL_LIST_FILE
      • text file listing calibration images (filename only. no extension)
    • CAL_BATCH_SIZE
      • batch size for calibration
    • CAL_NB_BATCHES
      • the number of batches
    • CAL_IMAGE_C
      • the channel of calibration image. must be the same as model
    • CAL_IMAGE_H
      • the height of calibration image. must be the same as model
    • CAL_IMAGE_W
      • the width of calibration image. must be the same as model
    • CAL_SCALE
      • normalize parameter for calibration (probably, should use the same value as trainig)
    • CAL_BIAS
      • normalize parameter for calibration (probably, should use the same value as trainig)

Quantization Calibration

  • If you want to use int8 mode, you need calibration step
  1. Create ppm images whose size is the same as model input size from training images
    • you can use inference_helper/tensorrt/calibration/batchPrepare.py
    • python .\batchPrepare.py --inDir sample_org --outDir sample_ppm
  2. Copy the generated ppm files and list.txt to the target environment such as Jetson
  3. Use .onnx model
  4. Modify parameters for calibration such as CAL_DIR and define USE_INT8
  5. Compile the project and run it
  6. If it succeeds, trt model file is generated. You can use it after that

Note

  • Install TensorRT in Windows
    • cuDNN installation
      • Copy all files into CUDA directory
    • TensorRT installation
      • Copy all files into CUDA directory
      • Or, set environment variable(TensorRT_ROOT = C:\Program Files\NVIDIA GPU Computing Toolkit\TensorRT\TensorRT-8.2.0.6), and add %TensorRT_ROOT%\lib to path

License

  • Copyright 2020 iwatake2222
  • Licensed under the Apache License, Version 2.0

Acknowledgements