Pano3D

This repo contains the source code for project's web page.

A Holistic Benchmark and a Solid Baseline for 360^o Depth Estimation

Pano3D is a new benchmark for depth estimation from spherical panoramas. We generate a dataset (using GibsonV2) and provide baselines for holistic performance assessment, offering:

Primary and secondary traits metrics:
- Direct depth performance:
  - (w)RMSE
  - (w)RMSLE
  - AbsRel
  - SqRel
  - (w)Relative accuracy (\delta) @ {1.05, 1.1, 1.25, 1.25², 1.25³ }
- Boundary discontinuity preservation:
  - Precision @ {0.25, 0.5, 1.0}m
  - Recall @ {0.25, 0.5, 1.0}m
  - Depth boundary errors of accuracy and completeness
- Surface smoothness:
  - RMSE^o
  - Relative accuracy (\alpha) @ {11.25^o, 22.5^o, 30^o}
Out-of-distribution & Zero-shot cross dataset transfer:
- Different depth distribution test set
- Varying scene context test set
- Shifted camera domain test set

By disentangling generalization and assessing all depth properties, Pano3D aspires to drive progress benchmarking for 360^o depth estimation.

Using Pano3D to search for a solid baseline results in an acknowledgement of exploiting complementary error terms, adding encoder-decoder skip connections and using photometric augmentations.

TODO

Demo

A publicly hosted demo of the baseline models can be found here. Using the web app, it is possible to upload a panorama and download a 3D reconstructed mesh of the scene using the derived depth map.

Note that due to the external host's caching issues, it might be necessary to refresh your browser's cache in between runs to update the 3D models.

The model's code and weights are also available in our demo.

Data

Download

To download the data, follow the instructions at vcl3d.github.io/Pano3D/download/.

Please note that getting access to the data download links is a two step process as the dataset is a derivative and compliance with the original dataset's terms and usage agreements is required. Therefore: