Note 1: r3live requires OpenCV >= 3.3. Use this command to check your opencv version:
pkg-config --modversion opencv
To switch ROS OpenCV version to 3.4.16 for r3live compatibility, do the following:
mkdir ~/third-party-library-folder && cd ~/third-party-library-folder
git clone -b 3.4.16 https://github.com/opencv/opencv_contrib.git
git clone -b 3.4.16 https://github.com/opencv/opencv.git
cd opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D WITH_TBB=ON \-D WITH_V4L=ON \-D WITH_QT=ON \-D WITH_OPENGL=ON \
-D WITH_CUDA=ON \-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
-D ENABLE_CXX11=ON \
-D CMAKE_C_COMPILER=/usr/bin/gcc-5 \
-D BUILD_EXAMPLES=OFF ..
make -j $(nproc)
sudo make install
Restart your computer afterwards...
The lines find_package(OpenCV 3.4.16 REQUIRED) and set(OpenCV_DIR $HOME/third-party-library-folder/opencv/build) were added to the CMakeList.txt file so that catkin uses the correct OpenCV version when building r3live. Note that you may still encounter runtime errors even if the build passes if your opencv version is not >= 3.3.
Note 2: If you encounter an error during build that has to do with flann and includes the following line:
/usr/include/flann/ext/lz4.h:249:72: error: conflicting declaration ‘typedef struct LZ4_streamDecode_t LZ4_streamDecode_t’
typedef struct { unsigned long long table[LZ4_STREAMDECODESIZE_U64]; } LZ4_streamDecode_t;
Then run the following commands to fix the LZ4 symbolic links (taken from here: ):
sudo mv /usr/include/flann/ext/lz4.h /usr/include/flann/ext/lz4.h.bak
sudo mv /usr/include/flann/ext/lz4hc.h /usr/include/flann/ext/lz4.h.bak
sudo ln -s /usr/include/lz4.h /usr/include/flann/ext/lz4.h
sudo ln -s /usr/include/lz4hc.h /usr/include/flann/ext/lz4hc.h
Note 3: The r3live code was modified to publish the VIO pose of the camera to the camera_odom frame.
A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package
[Jan 28, 2022] Release of our hardware design: The CAD files of our hardware design are now available in rxlive_handheld. You can download, print, edit and assemble our handheld device by following the guides.
[Jan 14, 2022] Add support of spinning LiDAR: Give an example of launching R3LIVE with a spinning LIDAR (see section 7).
R3LIVE is a novel LiDAR-Inertial-Visual sensor fusion framework, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE is built upon our previous work R2LIVE, is contained of two subsystems: the LiDAR-inertial odometry (LIO) and the visual-inertial odometry (VIO). The LIO subsystem (FAST-LIO) takes advantage of the measurement from LiDAR and inertial sensors and builds the geometric structure of (i.e. the position of 3D points) global maps. The VIO subsystem utilizes the data of visual-inertial sensors and renders the map's texture (i.e. the color of 3D points).
Our preprint paper is available here, with our accompanying videos are now available on YouTube (click below images to open) and Bilibili1, 2.
R3LIVE is robust enough to work well in various of LiDAR-degenerated scenarios (see following figures):
And even in simultaneously LiDAR degenerated and visual texture-less environments (see Experiment-1 of our paper).
R3LIVE is able to reconstruct the precise, dense, 3D, RGB-colored maps of surrounding environment in real-time (watch this video).
To make R3LIVE more extensible, we also provide a series of offline utilities for reconstructing and texturing meshes, which further reduce the gap between R3LIVE and various 3D applications (watch this video).
Following this ROS Installation to install ROS and its additional pacakge:
sudo apt-get install ros-XXX-cv-bridge ros-XXX-tf ros-XXX-message-filters ros-XXX-image-transport ros-XXX-image-transport*
NOTICE: remember to replace "XXX" on above command as your ROS distributions, for example, if your use ROS-kinetic, the command should be:
sudo apt-get install ros-kinetic-cv-bridge ros-kinetic-tf ros-kinetic-message-filters ros-kinetic-image-transport*
Follow this livox_ros_driver Installation.
sudo apt-get install libcgal-dev pcl-tools
You can use the following command to check your OpenCV version, if your openCV version lower than OpenCV-3.3, we recommend you to update your you openCV version if you meet errors in complying our codes. Otherwise, skip this step ^_^
pkg-config --modversion opencv
We have successfully test our algorithm with version 3.3.1, 3.4.16, 4.2.1 and 4.5.3.
Notice: We have noticed that a large number of users meet a crash of problem after launching our package due to the mismatch of openCV version (see issue #11, issue #20 and issue #23, and etc.). If you meet with similar problems, please make sure that the OpenCV you complied are same as the OpenCV you run with. This is very very important to launch R3LIVE correctly.
Clone this repository and catkin_make:
cd ~/catkin_ws/src
git clone https://github.com/hku-mars/r3live.git
cd ../
catkin_make
source ~/catkin_ws/devel/setup.bash
5.1 Download our rosbag files (r3live_dataset)
Our datasets for evaluation can be download from our Google drive or Baidu-NetDisk [百度网盘] (code提取码: wwxw). We have released totally 9 rosbag files for evaluating r3live, with the introduction of these datasets can be found on this page.
After you have downloaded our bag files, you can now run our example ^_^
roslaunch r3live r3live_bag.launch
rosbag play YOUR_DOWNLOADED.bag
If everything is correct, you will get the result that matches our paper and the results posted on this page.
R3LIVE allow you to save the maps you build at anytime you wanted. You just need to click on the "Control panel" and press 'S' or 's' key.
After you have save your offline map on your disk (default save in directory: ${HOME}/r3live_output), you can launch our utility to reconstruct and texture your mesh.
roslaunch r3live r3live_reconstruct_mesh.launch
As default, your offline map (and reconstructed mesh) will be saved in the directory ${HOME}/r3live_output, you can open it with pcl_viewer (and meshlab).
Install pcl_viewer and meshlab:
sudo apt-get install pcl-tools meshlab
Visualizing your offline point cloud maps (with suffix *.pcd):
cd ${HOME}/r3live_output
pcl_viewer rgb_pt.pcd
Visualizing your reconstructed mesh (with suffix *.ply):
cd ${HOME}/r3live_output
meshlab textured_mesh.ply
Since the LiDAR data and IMU data published by the official Livox-ros-driver is with the timestamp of LiDAR (started from 0 in each recording), and the timestamp of the image is usually recorded with the timestamp of the operation system. To make them working under the same time-based, we modified the source code of Livox-ros-driver, which is available at here. We suggest you replace the official driver with it when sampling your own data for R3LIVE.
In order to launch R3LIVE on your own hardware setup, you need to have a carefully calibration of the extrinsic among LiDAR, Camera and IMU. We recommend you using the following repo to kindly calibrate your sensors:
livox_camera_calib: A robust, high accuracy extrinsic calibration tool between high resolution LiDAR (e.g. Livox) and camera in targetless environment.
Even though our proposed method is unrelated to what kind of LiDAR you used, it is impossible for us making R3LIVE compatible with all kinds of existing LiDARs. To launch R3LIVE with spinning LIDAR, it requires you to take some effort in modifying the source code of our LiDAR front-end (see LiDAR_front_end.cpp). Here we give an example to test our LIO-subsystem with an Ouster-2 64 Line Spinning LIDAR.
Download our recorded rosbag file from here.
roslaunch r3live r3live_bag_ouster.launch
rosbag play ouster_example_for_LIO_test.bag
Notice: We manually disable our VIO-subsystem due the missed of calibration files in this example.
Finally, we are still working on making R3LIVE compatible with more spinning LiDARs. More and more examples will be released in the future.
In order to facilitate our users in reproducing our work, we also make our hardware design public available, where you can download all of our CAD source files in rxlive_handheld.
We know our packages might not totally stable in this stage, and we are keep working on improving the performance and reliability of our codes. So, if you have met any bug or problem, please feel free to draw an issue and I will respond ASAP.
For reporting our problems and bugs, please attach both your hardware and software environment if possible (printed by R3LIVE, see the following figure), which will be a great help for me in locating your problems.
In the development of R3LIVE, we stand on the shoulders of the following repositories:
- R2LIVE: A robust, real-time tightly-coupled multi-sensor fusion package.
- FAST-LIO: A computationally efficient and robust LiDAR-inertial odometry package.
- ikd-Tree: A state-of-art dynamic KD-Tree for 3D kNN search.
- livox_camera_calib: A robust, high accuracy extrinsic calibration tool between high resolution LiDAR (e.g. Livox) and camera in targetless environment.
- LOAM-Livox: A robust LiDAR Odometry and Mapping (LOAM) package for Livox-LiDAR.
- openMVS: A library for computer-vision scientists and especially targeted to the Multi-View Stereo reconstruction community.
- VCGlib: An open source, portable, header-only Visualization and Computer Graphics Library.
- CGAL: A C++ Computational Geometry Algorithms Library.
The source code is released under GPLv2 license.
We are still working on improving the performance and reliability of our codes. For any technical issues, please contact me via email Jiarong Lin < ziv.lin.ljrATgmail.com >.
If you use any code of this repo in your academic research, please cite at least one of our papers:
[1] Lin, Jiarong, and Fu Zhang. "R3LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightly-coupled state Estimation and mapping package."
[2] Xu, Wei, et al. "Fast-lio2: Fast direct lidar-inertial odometry."
[3] Lin, Jiarong, et al. "R2LIVE: A Robust, Real-time, LiDAR-Inertial-Visual tightly-coupled state Estimator and mapping."
[4] Xu, Wei, and Fu Zhang. "Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter."
[5] Cai, Yixi, Wei Xu, and Fu Zhang. "ikd-Tree: An Incremental KD Tree for Robotic Applications."
[6] Lin, Jiarong, and Fu Zhang. "Loam-livox: A fast, robust, high-precision LiDAR odometry and mapping package for LiDARs of small FoV."
For commercial use, please contact me < ziv.lin.ljrATgmail.com > and Dr. Fu Zhang < fuzhangAThku.hk >.