/PointMVSNet

Code for "Point-based Multi-view Stereo Network" (ICCV 2019 Oral) & "Visibility-aware Point-based Multi-view Stereo Network" (TPAMI)

Primary LanguagePythonOtherNOASSERTION

Point-based Multi-view Stereo Network & Visibility-aware Point-based Multi-view Stereo Network

Introduction

PointMVSNet is a deep point-based deep framework for multi-view stereo (MVS). PointMVSNet directly processes the target scene as point clouds and predicts the depth in a coarse-to-fine manner. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point.

VAPointMVSNet extends PointMVSNet with visibility-aware multi-view feature aggregations, which allows the network to aggregate multi-view appearance cues while taking into account occlusions.

If you find this project useful for your research, please cite:

@ARTICLE{ChenVAPMVSNet2020TPAMI,
  author={Chen, Rui and Han, Songfang and Xu, Jing and Su, Hao},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Visibility-Aware Point-Based Multi-View Stereo Network}, 
  year={2020},
  volume={},
  number={},
  pages={1-1},}

@InProceedings{ChenPMVSNet2019ICCV,
    author = {Chen, Rui and Han, Songfang and Xu, Jing and Su, Hao},
    title = {Point-based Multi-view Stereo Network},
    booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
    year = {2019}
}

How to use

Environment

The environment requirements are listed as follows:

  • Pytorch 1.0.1
  • CUDA 9.0
  • CUDNN 7.4.2
  • GCC5

Installation

  • Check out the source code

    git clone https://github.com/callmeray/PointMVSNet && cd PointMVSNet

  • Install dependencies

    bash install_dependencies.sh

  • Compile CUDA extensions

    bash compile.sh

Training

  • Download the preprocessed DTU training data from MVSNet and unzip it to data/dtu.

  • Train the network

    python pointmvsnet/train.py --cfg configs/dtu_wde3.yaml

    You could change the batch size in the configuration file according to your own pc.

Testing

  • Download the rectified images from DTU benchmark and unzip it to data/dtu/Eval.

  • Test with your own model

    python pointmvsnet/test.py --cfg configs/dtu_wde3.yaml

  • Test with the pretrained model

    python pointmvsnet/test.py --cfg configs/dtu_wde3.yaml TEST.WEIGHT outputs/dtu_wde3/model_pretrained.pth

Depth Fusion

PointMVSNet generates per-view depth map. We need to apply depth fusion tools/depthfusion.py to get the complete point cloud. Please refer to MVSNet for more details.