PatchMatch

PatchMatch-Stereo-Panorama, a fast dense reconstruction from 360° video images

Hartmut Surmann · Marc Thurow · Dominik Slomma · Niklas Digakis

⚠️PLEASE NOTE⚠️: This software version is no longer under development.
The successor can be found at: stella_vslam_dense ⬅️

Paper (arXiv) [doi](https://doi.org/10.1109/SSRR56537.2022.10018698)

Additional material for the paper: "PatchMatch-Stereo-Panorama, a fast dense reconstruction from 360° video images"

| Abstract — This work proposes a new method for real-time dense 3d reconstruction for common 360° action cams, which can be mounted on small scouting UAVs during USAR missions. The proposed method extends a feature based Visual monocular SLAM (OpenVSLAM, based on the popular ORB-SLAM) for robust long-term localization on equirectangular video input by adding an additional densification thread that computes dense correspondences for any given keyframe with respect to a local keyframe-neighboorhood using a PatchMatch-Stereo-approach. While PatchMatch-Stereo-types of algorithms are considered state of the art for large scale Mutli-View-Stereo they had not been adapted so far for real-time dense 3d reconstruction tasks. This work describes a new massivelly parallel variant of the PatchMatch-Stereo-algorithm that differs from current approaches in two ways:

First it supports the equirectangular camera model while other solutions are limited to the pinhole camera model. Second it is optimized for low latency while keeping a high level of completeness and accuracy. To achieve this it operates only on small sequences of keyframes but employs techniques to compensate for the potential loss of accuracy due to the limited number of frames. Results demonstrate that dense 3d reconstruction is possible on a consumer grade laptop with a recent mobile GPU and that it is possible with improved accuracy and completeness over common offline-MVS solutions with comparable quality settings.

Example of the UAVs with 360° camera

Keywords: PatchMatch-Stereo, 360°-Panorama, visual monocular SLAM, UAV, Rescue Robotics

Usage:

Requirements

NVIDIA Container Toolkit

Build

git clone https://github.com/RoblabWh/PatchMatch
cd ~/PathMatch
docker build -t pmdvslam --build-arg NUM_THREADS=$(nproc) .

Run

Start Container

xhost +local:
nvidia-docker run -it --rm -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix:ro -v /HOST/DATA/PATH:/data pmdvslam

Run VSLAM (press "Terminate" to export and exit)

./run_video_slam -v /PATH/TO/ORB_VOCAB.dbow2 -c /PATH/TO/CONFIG.yaml -m /PATH/TO/VIDEO [--mask /PATH/TO/MASK] [-p /PATH/TO/DATABASE.msg]

Export .msg to .ply (already included in normal .msg export)

python3 ./export_dense_msg_to_ply.py -i /PATH/TO/FILE.msg_dense -o /PATH/TO/OUTPUT.ply

Realtime Demo

Start Container and mount this repository as docker volume to /data

cd ~/PatchMatch
xhost +local:
nvidia-docker run -it --rm -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix:ro -v ${PWD}:/data pmdvslam

Run Demo

./run_video_slam -v /data/orb_vocab/orb_vocab.dbow2 -c /data/example/patchmatch/config.yaml -m /data/example/patchmatch/video.mp4 --mask /data/example/patchmatch/mask.png -p /data/example/patchmatch/output.msg --frame-skip 2 --no-sleep

Application Videos:

Cite:

@INPROCEEDINGS{10018698,
author={Surmann, Hartmut and Thurow, Marc and Slomma, Dominik},
booktitle={2022 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)},
title={PatchMatch-Stereo-Panorama, a fast dense reconstruction from 360° video images},
year={2022},
volume={},
number={},
pages={366-372},
doi={10.1109/SSRR56537.2022.10018698}}

RoblabWh/PatchMatch