RELLIS-3D: A Multi-modal Dataset for Off-Road Robotics
Peng Jiang1, Philip Osteen2, Maggie Wigness2 and Srikanth Saripalli1
1. Texas A&M University; 2. CCDC Army Research Laboratory
[Website] [Paper] [Github]
Updates
- 11/26/2020 v1.0 release
- 02/25/2021 improve camera and lidar calibration parameter
- 03/04/2021 update ROS bag with new tf (v1.1 release)
- 06/14/2021 fix missing labels of point cloud and fix wrong poses
- 01/24/2022 add Velodyne point clouds in kitti format and labels transfered from Ouster
Overview
Semantic scene understanding is crucial for robust and safe autonomous navigation, particularly so in off-road environments. Recent deep learning advances for 3D semantic segmentation rely heavily on large sets of training data; however, existing autonomy datasets represent urban environments or lack multimodal off-road data. We fill this gap with RELLIS-3D, a multimodal dataset collected in an off-road environment containing annotations for 13,556 LiDAR scans and 6,235 images. The data was collected on the Rellis Campus of Texas A&M University and presents challenges to existing algorithms related to class imbalance and environmental topography. Additionally, we evaluate the current state of the art deep learning semantic segmentation models on this dataset. Experimental results show that RELLIS-3D presents challenges for algorithms designed for segmentation in urban environments. Except for the annotated data, the dataset also provides full-stack sensor data in ROS bag format, including RGB camera images, LiDAR point clouds, a pair of stereo images, high-precision GPS measurement, and IMU data. This novel dataset provides the resources needed by researchers to develop more advanced algorithms and investigate new research directions to enhance autonomous navigation in off-road environments.
Recording Platform
Sensor Setup
- 64 channels Lidar: Ouster OS1
- 32 Channels Lidar: Velodyne Ultra Puck
- 3D Stereo Camera: Nerian Karmin2 + Nerian SceneScan (Sensor Configuration)
- RGB Camera: Basler acA1920-50gc + Edmund Optics 16mm/F1.8 86-571
- Inertial Navigation System (GPS/IMU): Vectornav VN-300 Dual Antenna GNSS/INS
Folder structure
Rellis-3D
├── pt_test.lst
├── pt_val.lst
├── pt_train.lst
├── pt_test.lst
├── pt_train.lst
├── pt_val.lst
├── 00000
├── os1_cloud_node_kitti_bin/ -- directory containing ".bin" files with Ouster 64-Channels point clouds.
├── os1_cloud_node_semantickitti_label_id/ -- containing, ".label" files for Ouster Lidar point cloud with manually labelled semantics label
├── vel_cloud_node_kitti_bin/ -- directory containing ".bin" files with Velodyne 32-Channels point clouds.
├── vel_cloud_node_semantickitti_label_id/ -- containing, ".label" files for Velodyne Lidar point cloud transfered from Ouster point cloud.
├── pylon_camera_node/ -- directory containing ".png" files from the color camera.
├── pylon_camera_node_label_color -- color image lable
├── pylon_camera_node_label_id -- id image lable
├── calib.txt -- calibration of velodyne vs. camera. needed for projection of point cloud into camera.
└── poses.txt -- file containing the poses of every scan.
Download Link on BaiDu Pan:
链接: https://pan.baidu.com/s/1akqSm7mpIMyUJhn_qwg3-w?pwd=4gk3 提取码: 4gk3 复制这段内容后打开百度网盘手机App，操作更方便哦
Annotated Data:
Ontology:
With the goal of providing multi-modal data to enhance autonomous off-road navigation, we defined an ontology of object and terrain classes, which largely derives from the RUGD dataset but also includes unique terrain and object classes not present in RUGD. Specifically, sequences from this dataset includes classes such as mud, man-made barriers, and rubble piles. Additionally, this dataset provides a finer-grained class structure for water sources, i.e., puddle and deep water, as these two classes present different traversability scenarios for most robotic platforms. Overall, 20 classes (including void class) are present in the data.
Ontology Definition (Download 18KB)
Images Statics:
Note: Due to the limitation of Google Drive, the downloads might be constrained. Please wait for 24h and try again. If you still can't access the file, please email maskjp@tamu.edu with the title "RELLIS-3D Access Request"..
Image Download:
Image with Annotation Examples (Download 3MB)
Full Images (Download 11GB)
Full Image Annotations Color Format (Download 119MB)
Full Image Annotations ID Format (Download 94MB)
Image Split File (44KB)
LiDAR Scans Statics:
LiDAR Download:
Ouster LiDAR with Annotation Examples (Download 24MB)
Ouster LiDAR with Color Annotation PLY Format (Download 26GB)
The header of the PLY file is described as followed:
element vertex
property float x
property float y
property float z
property float intensity
property uint t
property ushort reflectivity
property uchar ring
property ushort noise
property uint range
property uchar label
property uchar red
property uchar green
property uchar blue
To visualize the color of the ply file, please use CloudCompare or Open3D. Meshlab has problem to visualize the color.
Ouster LiDAR SemanticKITTI Format (Download 14GB)
To visualize the datasets using the SemanticKITTI tools, please use this fork: https://github.com/unmannedlab/point_labeler
Ouster LiDAR Annotation SemanticKITTI Format (Download 174MB)
Ouster LiDAR Scan Poses files (Download 174MB)
Ouster LiDAR Split File (75KB)
Velodyne LiDAR SemanticKITTI Format (Download 5.58GB)
Velodyne LiDAR Annotation SemanticKITTI Format (Download 143.6MB)
Calibration Download:
Camera Instrinsic (Download 2KB)
Basler Camera to Ouster LiDAR (Download 3KB)
Velodyne LiDAR to Ouster LiDAR (Download 3KB)
Stereo Calibration (Download 3KB)
Calibration Raw Data (Download 774MB)
Benchmarks
Image Semantic Segmenation
| models | sky | grass | tr ee | bush | concrete | mud | person | puddle | rubble | barrier | log | fence | vehicle | object | pole | water | asphalt | building | mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HRNet+OCR | 96.94 | 90.20 | 80.53 | 76.76 | 84.22 | 43.29 | 89.48 | 73.94 | 62.03 | 54.86 | 0.00 | 39.52 | 41.54 | 46.44 | 9.51 | 0.72 | 33.25 | 4.60 | 48.83 |
| GSCNN | 97.02 | 84.95 | 78.52 | 70.33 | 83.82 | 45.52 | 90.31 | 71.49 | 66.03 | 55.12 | 2.92 | 41.86 | 46.51 | 54.64 | 6.90 | 0.94 | 44.18 | 11.47 | 50.13 |
LiDAR Semantic Segmenation
| models | sky | grass | tr ee | bush | concrete | mud | person | puddle | rubble | barrier | log | fence | vehicle | object | pole | water | asphalt | building | mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SalsaNext | - | 64.74 | 79.04 | 72.90 | 75.27 | 9.58 | 83.17 | 23.20 | 5.01 | 75.89 | 18.76 | 16.13 | 23.12 | - | 56.26 | 0.00 | - | - | 40.20 |
| KPConv | - | 56.41 | 49.25 | 58.45 | 33.91 | 0.00 | 81.20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.40 | 0.00 | - | 0.00 | 0.00 | - | - | 18.64 |
Benchmark Reproduction
To reproduce the results, please refer to here
ROS Bag Raw Data
Data included in raw ROS bagfiles:
| Topic Name | Message Tpye | Message Descriptison |
|---|---|---|
| /img_node/intensity_image | sensor_msgs/Image | Intensity image generated by ouster Lidar |
| /img_node/noise_image | sensor_msgs/Image | Noise image generated by ouster Lidar |
| /img_node/range_image | sensor_msgs/Image | Range image generated by ouster Lidar |
| /imu/data | sensor_msgs/Imu | Filtered imu data from embeded imu of Warthog |
| /imu/data_raw | sensor_msgs/Imu | Raw imu data from embeded imu of Warthog |
| /imu/mag | sensor_msgs/MagneticField | Raw magnetic field data from embeded imu of Warthog |
| /left_drive/status/battery_current | std_msgs/Float64 | |
| /left_drive/status/battery_voltage | std_msgs/Float64 | |
| /mcu/status | warthog_msgs/Status | |
| /nerian/left/camera_info | sensor_msgs/CameraInfo | |
| /nerian/left/image_raw | sensor_msgs/Image | Left image from Nerian Karmin2 |
| /nerian/right/camera_info | sensor_msgs/CameraInfo | |
| /nerian/right/image_raw | sensor_msgs/Image | Right image from Nerian Karmin2 |
| /odometry/filtered | nav_msgs/Odometry | A filtered local-ization estimate based on wheel odometry (en-coders) and integrated IMU from Warthog |
| /os1_cloud_node/imu | sensor_msgs/Imu | Raw imu data from embeded imu of Ouster Lidar |
| /os1_cloud_node/points | sensor_msgs/PointCloud2 | Point cloud data from Ouster Lidar |
| /os1_node/imu_packets | ouster_ros/PacketMsg | Raw imu data from Ouster Lidar |
| /os1_node/lidar_packets | ouster_ros/PacketMsg | Raw lidar data from Ouster Lidar |
| /pylon_camera_node/camera_info | sensor_msgs/CameraInfo | |
| /pylon_camera_node/image_raw | sensor_msgs/Image | |
| /right_drive/status/battery_current | std_msgs/Float64 | |
| /right_drive/status/battery_voltage | std_msgs/Float64 | |
| /tf | tf2_msgs/TFMessage | |
| /tf_static | tf2_msgs/TFMessage | |
| /vectornav/GPS | sensor_msgs/NavSatFix | INS data from VectorNav-VN300 |
| /vectornav/IMU | sensor_msgs/Imu | Imu data from VectorNav-VN300 |
| /vectornav/Mag | sensor_msgs/MagneticField | Raw magnetic field data from VectorNav-VN300 |
| /vectornav/Odom | nav_msgs/Odometry | Odometry from VectorNav-VN300 |
| /vectornav/Pres | sensor_msgs/FluidPressure | |
| /vectornav/Temp | sensor_msgs/Temperature | |
| /velodyne_points | sensor_msgs/PointCloud2 | PointCloud produced by the Velodyne Lidar |
| /warthog_velocity_controller/cmd_vel | geometry_msgs/Twist | |
| /warthog_velocity_controller/odom | nav_msgs/Odometry |
ROS Bag Download
The following are the links for the ROS Bag files.
- Synced data (60 seconds example 2 GB): includes synced /os1_cloud_node/points, /pylon_camera_node/camera_info and /pylon_camera_node/image_raw
- Full-stack Merged data:(60 seconds example 4.2 GB): includes all data in above table and extrinsic calibration info data embedded in the tf tree.
- Full-stack Split Raw data:(60 seconds example 4.3 GB): is orignal data recorded by
rosbag recordcommand.
Sequence 00000: Synced data: (12GB) Full-stack Merged data: (23GB) Full-stack Split Raw data: (29GB)
Sequence 00001: Synced data: (8GB) Full-stack Merged data: (16GB) Full-stack Split Raw data: (22GB)
Sequence 00002: Synced data: (14GB) Full-stack Merged data: (28GB) Full-stack Split Raw data: (37GB)
Sequence 00003:Synced data: (8GB) Full-stack Merged data: (15GB) Full-stack Split Raw data: (19GB)
Sequence 00004:Synced data: (7GB) Full-stack Merged data: (14GB) Full-stack Split Raw data: (17GB)
ROS Environment Installment
The ROS workspace includes a plaftform description package which can provide rough tf tree for running the rosbag.
To run cartographer on RELLIS-3D please refer to here
Full Data Download:
Citation
@misc{jiang2020rellis3d,
title={RELLIS-3D Dataset: Data, Benchmarks and Analysis},
author={Peng Jiang and Philip Osteen and Maggie Wigness and Srikanth Saripalli},
year={2020},
eprint={2011.12954},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Collaborator
License
All datasets and code on this page are copyright by us and published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Related Work
SemanticUSL: A Dataset for Semantic Segmentation Domain Adatpation
LiDARNet: A Boundary-Aware Domain Adaptation Model for Lidar Point Cloud Semantic Segmentation
A RUGD Dataset for Autonomous Navigation and Visual Perception inUnstructured Outdoor Environments