Spherical View Synthesis for Self-Supervised 360^o Depth Estimation

Data

The 360^o stereo data used to train the self-supervised models are available here and are part of a larger dataset [1, 2] that contains rendered color images, depth and normal maps for each viewpoint in a trinocular setup.

Train

Training code to reproduce our experiments is available in this repository:

A set of training scripts are available for each different variant:

train_ud.py for vertical stereo (UD) training
train_lr.py for horizontal stereo (LR) training
train_tc.py for trinocular stereo (TC) training, using the photo_ratio argument to train the different TC variants.
train_sv.py for supervised (SV) training

The PyTorch implementation of the differentiable depth-image-based forward rendering (splatting), presented in [3] and originally implemented in TensorFlow, is also available.

Test

Our evaluation script test.py also includes the adaptation of the metrics calculation to spherical data that includes spherical weighting and spiral sampling.

Pre-trained Models

Our PyTorch pre-trained models (corresponding to those reported in the paper) are available at our releases and contain these model variants:

Citation

If you use this code and/or data, please cite the following:

@inproceedings{zioulis2019spherical,
  author       = "Zioulis, Nikolaos and Karakottas, Antonis and Zarpalas, Dimitris and Alvarez, Federic and Daras, Petros",
  title        = "Spherical View Synthesis for Self-Supervised $360^o$ Depth Estimation",
  booktitle    = "International Conference on 3D Vision (3DV)",
  month        = "September",
  year         = "2019"
}

References

[1] Zioulis, N.*, Karakottas, A.*, Zarpalas, D., and Daras, P. (2018). Omnidepth: Dense depth estimation for indoors spherical panoramas. In Proceedings of the European Conference on Computer Vision (ECCV).

[2] Karakottas, A., Zioulis, N., Samaras, S., Ataloglou, D., Gkitsas, V., Zarpalas, D., and Daras, P. (2019). 360^o Surface Regression with a Hyper-sphere Loss. In Proceedings of the International Conference on 3D Vision (3DV).

[3] Tulsiani, S., Tucker, R., and Snavely, N. (2018). Layer-structured 3d scene inference via view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV).

nicholaspei/SphericalViewSynthesis

Spherical View Synthesis for Self-Supervised 360o Depth Estimation