A TensorRT ROS2 package for realtime instance segmentation in C++ using fine-tuned YOLOv8. This package is designed to be used with ROS2 Iron and Ubuntu 22.04.
This packages assumes you have already fine-tuned a YOLOv8 model and have the ONNX model file.
Note: This topic will only be published if the
visualizeMasks
parameter is set totrue
.
This topic is a sensor_msgs/Image
image and is used to visualize the output of the instance segmentation and bounding boxes in RViz2.
- The
Header
(with timestamp
andframe_id
),height
,width
,is_bigendian
, andstep
are taken from the original image topic that the mask is associated with and theencoding
of the image which is alwaysbgr8
.
This topic publishes a custom yolov8_interfaces/msg/Detection
message which contains the class label, class probability, class name, segmentation mask, and bounding box for each detected object. Everything in this message is in the same order as the detected objects in the image. For example, the second element in the labels
list corresponds to the second binary mask in the masks
list and the second bounding box in the bounding_boxes
list.
index
- an int representing the index of the detected object (1 indexed since 0 is reserved for the background)labels
- a list of class labels (int) for each detected objectprobabilities
- a list of class probabilities (float range 0 - 1) for each detected objectclass_names
- a list of human readable class names (string) for each detected objectseg_mask_one_channel
- a single channel segmentation mask for all detected segmentation masks combined into one image and in one color channel- Each mask (Image) is a 2D array where 0 represents the background and all other values represent the
index
of the detected object. - Published as a
sensor_msgs/Image
image - The
Header
(with timestamp
andframe_id
),height
,width
,is_bigendian
, andstep
are taken from the original image topic that the mask is associated with and theencoding
of the image which is alwaysmono8
. - This topic is useful for tasks like LiDAR point cloud segmentation where you want to segment the point cloud based on the projected detections from image space onto the point cloud.
data
is the image itself
- Each mask (Image) is a 2D array where 0 represents the background and all other values represent the
bounding_boxes
- a list of bounding boxes for each detected object- Defined as a custom message
yolov8_interfaces/msg/Yolov8BBox.msg
top_left
- a custom messageyolov8_interfaces/msg/Point2D.msg
representing the top left corner of the bounding box as intx
andy
coordinatesrect_width
- the width of the bounding box as an intrect_height
- the height of the bounding box as an int
- Defined as a custom message
Note: This topic will only be published if the
enableOneChannelMask
andvisualizeOneChannelMask
parameters are both set totrue
.
This topic is a sensor_msgs/Image
image and is used to visualize the output oneChannelMask (all the segmentation masks combined into one image and in one color) in RViz2 / Foxglove Studio.
- The
Header
(with timestamp
andframe_id
),height
,width
,is_bigendian
, andstep
are taken from the original image topic that the mask is associated with. The published imageencoding
is actually anrgb8
converted from the originalmono8
yolov8/camera_name/detections/seg_mask_one_channel
topic. - The
rgb8
image is normalized using cv::NORMALIZE_MINMAX so the minimum value of the displayed image is 0 and the maximum value of the displayed image is 255 even if the original image only has 0 and 1 values. This means the labels will change colors drastically if the number of detected objects changes. This is done so the image can be visualized in RViz2 / Foxglove Studio. data
is the image itself
-
Made for Ubuntu 22.04 (Untested on other Distributions)
-
Install ROS2 Iron
- Since ROS2 Iron natively supports Ubuntu 22.04 you can install is as a Debian package as described here.
-
Make sure you are not using a conda environment as it may cause issues with the ROS2 environment.
-
Install CUDA 11.8 Toolkit via the runfile (local) here NOT the deb version.
- WARNING: After installing the CUDA toolkit make sure you can run
nvidia-smi
in the CLI and that it doesn't error out. If it errors out, this may be because you installed the deb version of CUDA 11.8 rather than the runfile version or you forgot the--toolkit
flag. This means you installed newer CUDA low level drivers in place of the original CUDA low level drivers and may black screen your computer if your computer is restarted. Open up theAdditional Drivers
tab in theSoftware & Updates
application in Ubuntu and install the NVIDIA driver metapackage with the(proprietary, tested)
tags (or whatever driver version your GPU needs).
- WARNING: After installing the CUDA toolkit make sure you can run
-
Download cuDNN 8.2.4 from here and install it by following the instructions here.
-
Install TensorRT. Note you can install TensorRT as a DEB or TAR pkg. The DEB package is easier to install but only allows one minor version of TensorRT to be installed at a time. The TAR package is more difficult to install but allows multiple versions of TensorRT to be installed at the same time. You only have to install either the DEB or TAR package, not both.
-
Install TensorRT 10 GA (DEB Package)
- Download TensorRT 10 DEB Package from here by following these instructions. Install the full C++ and Python runtimes.
- The DEB package is always installed in
/usr/src/tensorrt
and the shared libraries are in/usr/lib/x86_64-linux-gnu/
. The CMAKE file in this package assumes the DEB package is used so you don't have to make any changes to the CMAKE files.
-
Install TensorRT 8.6 GA (TAR Package)
- Download TensorRT 8.6 GA TAR Package from here, place / unpack the tar inside the
~/libs/
folder, and install the full runtime by following the instructions here. - Since we installed the TAR package in a custom location (~/libs/TensorRT-xxx) we need to update
src/yolov8/libs/tensorrt-cpp-api/CMakeLists.txt
to point to the correct location of the TensorRT shared libraries. Change theset(TENSORRT_DIR "/usr/src/tensorrt")
line toset(TensorRT_DIR $ENV{HOME}/libs/TensorRT-8.6.1.6/)
in theCMakeLists.txt
file.
- Download TensorRT 8.6 GA TAR Package from here, place / unpack the tar inside the
-
Build OpenCV 4.8.0 with CUDA support using the following
-
In
scripts/install_opencv.sh
change theCUDA_ARCH_BIN
variable to the correct CUDA architecture of your GPU. You can find the correct CUDA compute architecture for your GPU here by clicking the dropdowns. -
In the root directory of this package, run:
./scripts/install_opencv.sh
-
This script will build OpenCV 4.8.0 with CUDA support and install it in the
libs/tensorrt-cpp-api/scripts/
directory, which will take a while to build. -
Note: This script will install a custom build of OpenCV system-wide, overwriting any existing OpenCV installation. This may break other projects that rely on a different version of OpenCV as across OpenCV versions functions are deprecated, removed, or changed. If this is a concern, you can uninstall this custom build of OpenCV with the following command but this package will no longer work until you reinstall the custom build of OpenCV.
./scripts/uninstall_opencv.sh
-
First you must create a ROS2 workspace.
mkdir -p ros_workspace/src
Then move the yolov8
package (directory) and yolov8_interfaces
package from the src
directory of this project into the src
directory of the ROS2 workspace you just created.
Then in the root directory of this ros_workspace
, run the following commands to build the ROS2 package and source the setup file.
# Builds the package with debug symbols so you can debug C++ code with GDB
make build-debug PACKAGES="yolov8 yolov8_interfaces"
source install/setup.bash
You can also build normally with colcon and source the setup file.
colcon build
source install/setup.bash
Then you can run the ROS2 node using the following command where model.onnx
is the path to the ONNX model you want to use.
ros2 run yolov8 ros_segmentation --model model.onnx
You can also run the ROS2 node using a launch file where the arguments
should be modified to the path of the ONNX model you want to use.
ros2 launch yolov8 yolov8.launch.py
Note: The first time you run the ROS node, it will take a while for TensorRT to build the engine model for your specific GPU. Subsequent runs will be able to load the serialized engine model and will be much faster.
You can also use the tmuxp
package to quickly run the ROS2 node in a tmux session. First install tmuxp
with the following command.
sudo apt install tmuxp
There are two tmuxp configurations in the tmuxp
directory.
rviz2_yolov8.yaml
- This configuration will run a configured RViz2, the ROS2 node, arosbag play
command, and a terminal to echo the topics.foxglove_yolov8.yaml
- This configuration will run Foxglove Studio, the ROS2 node, arosbag play
command, and a terminal to echo the topics.
In general if YOLOV8 is not working / crashing check the latest ros logs with the following command:
cat ~/.ros/log/latest.log
A list of common errors and their solutions are below:
- Exit code
-9
while building / loading the TensorRT Engine- Your computer likely ran out of memory causing the OOM Killer to kill the program. Try closing other applications or using a different GPU / or a computer with more memory. Use
htop
to monitor memory usage andnvidia-smi
to monitor GPU memory usage.
- Your computer likely ran out of memory causing the OOM Killer to kill the program. Try closing other applications or using a different GPU / or a computer with more memory. Use
- CUDA initialization failure with error: 46
- Make sure no other applications are taking control of the GPU. If you can't fix this issue, you can try restarting your computer.
- An error about no rclpy module check
- Make sure you are not using a conda environment as it may cause issues with the ROS2 environment (ROS2 does not officially support conda environments).
- An error when you launch / run the ros node saying
libnvinfer.so.8: cannot open shared object file: No such file or directory
- If you installed TensorRT as a TAR package, make sure you added the TensorRT path to the
LD_LIBRARY_PATH
environment variable in your~/.bashrc
file and sourced it as mentioned in Installation / Dependencies.
- If you installed TensorRT as a TAR package, make sure you added the TensorRT path to the
Error, cuda_runtime_api.h could not determine number of CUDA-capable devices. Restart your computer. Error thrown by cuda_runtime_api.h: unknown error
- This error seems to happen after the computer has been on for a long time and the GPU has been used a lot. Restarting the computer seems to fix this issue.
This project uses code and directions from YOLOv8-TensorRT-CPP licensed under the MIT license.
- Fix RViz2 Config
- Fix Makefile build targets
- Bring in CUDA Tools