RoMe: Towards Large Scale Road Surface Reconstruction via Mesh Representation

Paper,YouTube, Bilibili

Ruohong Mei, Wei Sui, Jiaxin Zhang, Xue Qin, Gang Wang, Tao Peng, Cong Yang

RoMe can also be applied for driving drowsiness reduction based on road surface element matching and alarming (Integration of Cabin and Driving). More details: https://fatigueview.github.io/

nuScenes

In configs/local_nusc.yaml

base_dir: Put official nuScenes here, e.g. {base_dir}/v1.0-trainval
image_dir: Put segmentation results here. We use Mask2Former to segment source images. Folder structure is like {image_dir}/{sweeps/samples}/seg_CAM_FRONT. We provide processed data on google drive, including all semantic images used in this paper. "Scene-1" and "Scene-2" of the paper contain scene-0063, scene-0064, scene-0200, scene-0283 and scene-0109, scene-0508, scene-0523, scene-0821 respectively.

KITTI Odom

In configs/local_kitti.yaml

base_dir: Put official kitti odometry dataset here, e.g. {base_dir}/sequences
image_dir: Put segmentation results here. Also we use Mask2Former and folder structure is like {image_dir}/seg_sequences. We provide processed data on google drive, including all semantic images and poses we used.

Quick Start

Environment

torch==1.10.2+cu111
torchvision==0.11.3+cu111
torchaudio==0.10.2+cu111
pytorch3d==0.6.1
pymeshlab==2021.10 
scipy opencv-py thon tqdm wandb python3.8

For wandb usage, please visit here.

Train a scene from nuScenes

Modify configs/local_nusc.yaml
- change wandb configs
- change base_dir and image_dir according to your folder
- change clip_list to train one scene or multiple scenes.
Modify wandb url and api_key in run_local.sh and then run sh run_local.sh to start training.

Train a scene from KITTI

Modify configs/local_kitti.yaml
- change wandb configs
- change base_dir and image_dir according to your folder
- change sequence to choose which sequence to train.
- modify choose_point and bev_x/y_length to choose which sub area to train.
Modify wandb url and api_key in run_local.sh and then run sh run_local.sh to start training.

Evaluation

Modify configs/nusc_eval.yaml
- change model_pathand pose_path where your trained models saved.
- Make sure other training parameters are same as your configs when training.
- This is a simple evaluation script and only support batch_size: 1

Future work

This is the first version of RoMe, and it is hard to reconstruct steep slopes.
Using SfM(structure from motion) or MVS(multi-view stereo) points and lidar points will give strong supervision.
Monodepth estimation would be useful like monosdf.
We are trying to use other mesh render methods like nvdiffrec.

Citation

@misc{mei2023rome,
      title={RoMe: Towards Large Scale Road Surface Reconstruction via Mesh Representation}, 
      author={Ruohong Mei and Wei Sui and Jiaxin Zhang and Xue Qin and Gang Wang and Tao Peng and Cong Yang},
      year={2023},
      eprint={2306.11368},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

DRosemei/RoMe