Authors: Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab and Federico Tombari
This is the SceneGraphFusion c++ framework. For the network part of the code, check here.
apt update
apt install git
# cmake
sudo apt install python3-pip
pip3 install cmake
# create ssh key and associate to your account in order to clone this project.
# * Generate new ssh key [link](https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent)
# * Adding a new ssh key to your account [link](https://docs.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account)
# Eigen3 3.3.90
# OpenCV 4.0.0
# This project will try to install them locally if they were not found.
loading meshes for online rendered view generation.
apt install libassimp-dev
With GUI:
# OpenGL & Glfw3
apt install freeglut3-dev libglfw3-dev
With graph prediction:
# Onnxruntime
# See https://github.com/microsoft/onnxruntime
build the basic segmentation system
git clone {this_repo}
cd SceneGraphFusion
git submodule init
git submodule update
mkdir build
cd build
cmake ..
make
build with GUI or graph prediction, pass these options in cmake:
cmake -DBUILD_GRAPHPRED=ON -DBUILD_GUI=ON ..
Note that the graph prediction module is not yet released. You may encounter error if you build with BUILD_GRAPHPRED=ON
. That part will be released soon.
./exe_GraphSLAM --pth_in path/to/3RScan/squence/
# or with GUI
./exe_GraphSLAM_GUI --pth_in path/to/3RScan/squence/
# to see usage and options
./exe_GraphSLAM --help
# to run with graph prediction (need to build with graph predict)
./exe_GraphSLAM_GUI --pth_in path/to/3RScan/squence/ --pth_model /path/to/the/traced/model/folder/
We provide data loader for 3RScan dataset and ScanNet. In default, we use rendered view for both dataset. The rendered view will be generated during on time.
you can download the traced model of our pretrained network here
To run our system with a test sequence, run bash prepare_example_data.sh
. It will download an example sequence provided from
3RScan repository. Then run
cd bin
./exe_GraphSLAM --pth_in ../data/3RScan/4acaebcc-6c10-2a2a-858b-29c7e4fb410d/sequence/
Or using the full sequence from either ScanNet or 3RScan
# For 3RScan:
./exe_GraphSLAM --pth_in [path/to/3RScan]/[sequence_id]/sequence/
# For ScanNet
./exe_GraphSLAM --pth_in [path/to/scannet]/[scan_id]/[scan_id].sens
# Run with graph prediction
./exe_GraphSLAM --pth_in ./path/to/data --pth_model /path/to/model
If you find the code useful please consider citing our paper:
@inproceedings{Wu2021,
title = {{SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences}},
author = {Shun-Cheng Wu and Johanna Wald and Keisuke Tateno and Nassir Navab and Federico Tombari},
booktitle = {Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2021}
}