/self-driving-ish_computer_vision_system

This project generates images you've probably seen in autonomous driving demo. Object Detection, Lane Detection, Road Segmentation, Depth Estimation using TensorRT

Primary LanguageC++Apache License 2.0Apache-2.0

very_short.mp4

Self-Driving-ish Computer Vision System

  • This project generates images you've probably seen in autonomous driving demo
  • Detection
    • Object Detection and Tracking
    • Lane Detection and Curve Fitting
    • Road Segmentation
    • Depth Estimation
  • Transform using Projective Geometry and Pinhole Camera Model
    • Normal View -> Top View
    • Distance Calculation (image plane -> ground plane in world coordinate system)

Result

result

Tested Environment

Computer

  • Windows 10 (x64) + Visual Studio 2019
    • Intel Core i7-6700 @ 3.4GHz + NVIDIA GeForce GTX 1070
  • Jetson Xavier NX. JetPack 4.6
    • You will get error if using JetPack 4.5 or before because of TensorRT error

Deep Learning Inference Framework

  • TensorFlow Lite with XNNPACK delegate
    • CPU
    • Note: Running with CPU is very slow
  • TensorRT
    • GPU

Usage

./main [input]
 - input:
    - use the default image file set in source code (main.cpp): blank
        - ./main
     - use video file: *.mp4, *.avi, *.webm
        - ./main test.mp4
     - use image file: *.jpg, *.png, *.bmp
        - ./main test.jpg
    - use camera: number (e.g. 0, 1, 2, ...)
        - ./main 0
    - use camera via gstreamer on Jetson: jetson
        - ./main jetson
  • Mouse Drag: Change top view angle
  • Keyboard (asdwzx) : Change top view position

How to build a project

0. Requirements

  • OpenCV 4.x
  • CMake
  • TensorRT 8.0.x
    • If you get build error related to TensorRT, modify cmake settings for it in inference_helper/inference_helper/CMakeLists.txt

1. Download source code and pre-built libraries

  • Download source code
    • If you use Windows, you can use Git Bash
    git clone https://github.com/iwatake2222/self-driving-ish_computer_vision_system.git
cd self-driving-ish_computer_vision_system
git submodule update --init
sh inference_helper/third_party/download_prebuilt_libraries.sh
  • Download models 
    sh ./download_resource.sh

2-a. Build in Windows (Visual Studio)

  • Configure and Generate a new project using cmake-gui for Visual Studio 2019 64-bit
    • Where is the source code : path-to-cloned-folder
    • Where to build the binaries : path-to-build (any)
  • Open main.sln
  • Set main project as a startup project, then build and run!
  • Note:
    • You may need to modify cmake setting for TensorRT for your environment

2-b. Build in Linux (Jetson Xavier NX)

mkdir build && cd build
# cmake .. -DENABLE_TENSORRT=off
cmake .. -DENABLE_TENSORRT=on
make
./main

Note

cmake options

cmake .. -DENABLE_TENSORRT=off  # Use TensorFlow Lite (default)
cmake .. -DENABLE_TENSORRT=on   # Use TensorRT

cmake .. -DENABLE_SEGMENTATION=on    # Enable Road Segmentation function (default)
cmake .. -DENABLE_SEGMENTATION=off   # Disable Road Segmentation function

cmake .. -DENABLE_DEPTH=on    # Enable Depth Estimation function (default)
cmake .. -DENABLE_DEPTH=off   # Disable Depth Estimation function

Misc

  • It will take very long time when you execute the app for the first time, due to model conversion
    • I took 80 minutes with RTX 3060ti
    • I took 10 - 20 minutes with GTX 1070

Software Design

Class Diagram

class_diagram

Data Flow Diagram

data_flow_diagram

Model Information

Details

Performance

Model Jetson Xavier NX GTX 1070
== Inference time ==
Object Detection 10.6 ms 6.4 ms
Lane Detection 9.6 ms 4.9 ms
Road Segmentation 29.1 ms 13.5 ms
Depth Estimation 55.2 ms 37.8 ms
== FPS ==
Total (All functions) 6.8 fps 10.9 fps
Total (w/o Segmentation, Depth) 24.4 fps 33.3 fps
  • Input
    • Jetson Xavier NX: Camera
    • GTX 1070: mp4 video
  • With TensorRT FP16
  • "Total" includes image read, pre/post process, other image process, result image drawing, etc.

License

  • Copyright 2021 iwatake2222
  • Licensed under the Apache License, Version 2.0

Acknowledgements

I utilized the following OSS in this project. I appreciate your great works, thank you very much.

Code, Library

Model

Image Files