Transformer-based Monocular Depth Estimation with Attention Supervision

Pretrained models

You can download pre-trained model

Trained with KITTI

cap a1 a2 a3 Abs Rel RMSE RMSE log

0-80m 0.963 0.995 0.999 0.058 2.685 0.089
Trained with NYU Depth V2

cap a1 a2 a3 Abs Rel log10 RMSE

0-10m 0.902 0.985 0.997 0.103 0.044 0.374

cap	a1	a2	a3	Abs Rel	RMSE	RMSE log
0-80m	0.963	0.995	0.999	0.058	2.685	0.089

cap	a1	a2	a3	Abs Rel	log10	RMSE
0-10m	0.902	0.985	0.997	0.103	0.044	0.374

Evaluation

Evaluate on KITTI

python3 eval.py --model_dir "PATH/KITTI.pkl" --evaluate --batch_size 1 --dataset KITTI --data_path "PATH" --gpu_num 0 --encoder AST

Evaluate on NYU2

python3 eval.py --model_dir "PATH/NYU2.pkl" --evaluate --batch_size 1 --dataset NYU --data_path "PATH" --gpu_num 0 --encoder AST

DPT-Large trained by us

python3 eval.py --model_dir "MODEL_PATH" --evaluate --batch_size 1 --dataset "DATASET" --data_path "PATH" --gpu_num 0 --other_method DPT-Large

Adabins trained by us

python3 eval.py --model_dir "MODEL_PATH" --evaluate --batch_size 1 --dataset "DATASET" --data_path "PATH" --gpu_num 0 --other_method Adabins

Dataset Preparation

We referred to LapDepth in the data preparation process. Thanks for sharing the processed dataset.

KITTI

1. Official ground truth

Download official KITTI ground truth on the link and make KITTI dataset directory.

    $ cd ./datasets
    $ mkdir KITTI && cd KITTI
    $ mv ~/Downloads/data_depth_annotated.zip ./datasets/KITTI
    $ unzip data_depth_annotated.zip

2. Raw dataset

Construct raw KITTI dataset using following commands.

    $ cd ./datasets/KITTI
    $ aria2c -x 16 -i ./datasets/kitti_archives_to_download.txt
    $ parallel unzip ::: *.zip

3. Dense g.t dataset
We take an inpainting method from DenseDepth to get dense g.t for gradient loss.
(You can train our model using only data loss without gradient loss, then you don't need dense g.t)
Corresponding inpainted results from './datasets/KITTI/data_depth_annotated/2011_xx_xx_drive_xxxx_sync/proj_depth/groundtruth/image_02' are should be saved in './datasets/KITTI/data_depth_annotated/2011_xx_xx_drive_xxxx_sync/dense_gt/image_02'.
KITTI data structures are should be organized as below:

|-- datasets
  |-- KITTI
     |-- data_depth_annotated  
        |-- 2011_xx_xx_drive_xxxx_sync
           |-- proj_depth  
              |-- groundtruth            # official G.T folder
        |-- ... (all drives of all days in the raw KITTI)  
     |-- 2011_09_26                      # raw RGB data folder  
        |-- 2011_09_26_drive_xxxx_sync
     |-- 2011_09_29
     |-- ... (all days in the raw KITTI)

NYU Depth V2

1. Training set
Make NYU dataset directory

    $ cd ./datasets
    $ mkdir NYU_Depth_V2 && cd NYU_Depth_V2

Constructing training data using following steps :
- Download Raw NYU Depth V2 dataset (450GB) from this Link.
- Extract the raw dataset into './datasets/NYU_Depth_V2'
  (It should make './datasets/NYU_Depth_V2/raw/....').
- Run './datasets/sync_project_frames_multi_threads.m' to get synchronized data. (need Matlab)
  (It shoud make './datasets/NYU_Depth_V2/sync/....').
Or, you can directly download whole 'sync' folder from our Google drive Link into './datasets/NYU_Depth_V2/'

2. Testing set
Download official nyu_depth_v2_labeled.mat and extract image files from the mat file.

    $ cd ./datasets
    ## Download official labled NYU_Depth_V2 mat file
    $ wget http://horatio.cs.nyu.edu/mit/silberman/nyu_depth_v2/nyu_depth_v2_labeled.mat
    ## Extract image files from the mat file
    $ python extract_official_train_test_set_from_mat.py nyu_depth_v2_labeled.mat splits.mat ./NYU_Depth_V2/official_splits/

WJ-Chang-42/ASTransformer