SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields

Anh-Quan Cao, Raoul de Charette
Inria, Paris, France.

If you find this work or code useful, please cite our paper and give this repo a star:

@InProceedings{cao2022scenerf,
    author    = {Cao, Anh-Quan and de Charette, Raoul},
    title     = {SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields},
    publisher = {arxiv},
    year      = {2022},
}

Teaser

Novel depths synthesis	3D Reconstruction

Table of Content

News
Installation
Datasets
Training
Evaluation
Acknowledgment

News

Pretrained models will be released soon 🚨

Installation

Create conda environment:

$ conda create -y -n scenerf python=3.7
$ conda activate scenerf

This code was implemented with python 3.7, pytorch 1.7.1 and CUDA 10.2. Please install PyTorch:

$ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch

Install the dependencies:

$ cd scenerf/
$ pip install -r requirements.txt

Install tbb

$ conda install -c bioconda tbb=2020.2

Downgrade torchmetrics

$ pip install torchmetrics==0.6.0

Finally, install scenerf:

$ pip install -e ./

Dataset

To train and evaluate novel depths/views synthesis, please download on KITTI Odometry website the following data:
- Odometry data set (calibration files, 1 MB)
- Odometry data set (color, 65 GB)
- Odometry ground truth poses (4 MB)
- Velodyne laser data, 80 GB
To evaluate scene reconstruction, please download the SemanticKITTI voxel data (700 MB).
Create a folder to store preprocess data at /path/to/kitti/preprocess/folder.
Store paths in environment variables for faster access (Note: folder 'dataset' is in /path/to/kitti):
```
$ export KITTI_PREPROCESS=/path/to/kitti/preprocess/folder
$ export KITTI_ROOT=/path/to/kitti 
```

Training

Create folders to store training logs at /path/to/kitti/logdir.

Store in an environment variable:

$ export KITTI_LOG=/path/to/kitti/logdir

Train scenerf using 4 v100-32g GPUs with batch_size of 4 (1 item per GPU):

$ cd scenerf/
$ python scenerf/scripts/train.py \
    --bs=4 --n_gpus=4 \
    --enable_log=True \
    --preprocess_root=$KITTI_PREPROCESS \
    --root=$KITTI_ROOT \
    --logdir=$KITTI_LOG \
    --n_gaussians=4 --n_pts_per_gaussian=8  \
    --max_epochs=50 --exp_prefix=Train

Evaluation

Create folders to store intermediate evaluation data at /path/to/evaluation/save/folder and reconstruction data at /path/to/reconstruction/save/folder.

$ export EVAL_SAVE_DIR=/path/to/evaluation/save/folder
$ export RECON_SAVE_DIR=/path/to/reconstruction/save/folder

Evaluate novel depths synthesis

Supposed we obtain the model from the training step at /path/to/model/checkpoint/last.ckpt. We follow the steps below to evaluate the novel depths synthesis performance.

Compute the depth metrics on all frames in each sequence, additionally grouped by the distance to the input frame.

$ cd scenerf/
$ python scenerf/scripts/evaluation/save_depth_metrics.py \
    --eval_save_dir=$EVAL_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --model_path=/path/to/model/checkpoint/last.ckpt

Aggregate the depth metrics from all sequences.

$ cd scenerf/
$ python scenerf/scripts/evaluation/agg_depth_metrics.py \
    --eval_save_dir=$EVAL_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS

Evaluate novel views synthesis

Given the trained model at /path/to/model/checkpoint/last.ckpt, the novel views synthesis performance is obtained as followed:

Render an RGB image for every frame in each sequence.

$ cd scenerf/
$ python scenerf/scripts/evaluation/render_colors.py \
    --eval_save_dir=$EVAL_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --model_path=/path/to/model/checkpoint

Compute the metrics, additionally grouped by the distance to the input frame.

$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_color.py --eval_save_dir=$EVAL_SAVE_DIR

Scene reconstruction

Generate novel views/depths for reconstructing scene.

$ cd scenerf/
$ python scenerf/scripts/reconstruction/generate_novel_depths.py \
    --recon_save_dir=$RECON_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --model_path=/path/to/model/checkpoint \
    --angle=10 --step=0.5 --max_distance=10.1

Convert the novel views/depths to TSDF volume. Note: the angle, step, and max_distance should match the previous step.

$ cd scenerf/
$ python scenerf/scripts/reconstruction/depth2tsdf.py \
    --recon_save_dir=$RECON_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --angle=10 --step=0.5 --max_distance=10.1

Compute scene reconstruction metrics using the generated TSDF volumes.

$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_sr.py \
    --recon_save_dir=$RECON_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS

Acknowledgment

The work was partly funded by French project SIGHT (ANR-20-CE23-0016) and was performed using HPC resources from GENCI–IDRIS (Grant 2021-AD011012808 and 2022-AD011012808R1). We thank Fabio Pizzati and Ivan Lopes for their kind proofreading.

mfahes/SceneRF