Focus On Depth - A single DPT encoder for AutoFocus application and Dense Prediction Tasks

Abstract

Depth estimation is a classic task in computer vision, which is of great significance for many applications such as augmented reality, target tracking and autonomous driving. We firstly summarize the deep learning models for monocular depth estimation. Secondly, we will implement a recent Vision Transformers based architecture for this task. We will seek to improve it by adding a segmentation head in order to perform multi-task learning using a customly built dataset. Thirdly, we will implement our model for in-the-wild images (i.e. without control on the environment, the distance and size of objects of interests, and their physical properties (rotation, dynamics, etc.)) for Auto-focus application on humans and will give qualitative comparison across other methods.

⚡ New! Web demo

You can check the webdemo hosted on Hugging Face and powered by Gradio, here.

📌 Requirements

Run: pip install -r requirements.txt

🚀 Running the model

You can first download one of the models from the model zoo:

🏦 Model zoo

Get the links of the following models:

FocusOnDepth_vit_base_patch16_384.p
Other models coming soon...

And put the .p file into the directory models/. After that, you need to update the config.json (Tutorial here) according to the pre-trained model you have chosen to run the predictions (this means that if you load a depth-only model, then you have to set type to depth for example ...).

🎯 Run a prediction

Put your input images (that have to be .png or .jpg) into the input/ folder. Then, just run python run.py and you should get the depth maps as well as the segmentation masks in the output/ folder.

🔨 Training

🔧 Build the dataset

Our model is trained on a combination of

📝 Configure `config.json`

Please refer to our config wiki to understand how to modify the config file to run a training.

🔩 Run the training script

After that, you can simply run the training script: python train.py

📜 Citations

Our work is based on the work from Ranflt et al. please do not forget to cite their work! :) You can also check our report if you need more details.

@article{DPT,
  author    = {Ren{\'{e}} Ranftl and
               Alexey Bochkovskiy and
               Vladlen Koltun},
  title     = {Vision Transformers for Dense Prediction},
  journal   = {CoRR},
  volume    = {abs/2103.13413},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.13413},
  eprinttype = {arXiv},
  eprint    = {2103.13413},
  timestamp = {Wed, 07 Apr 2021 15:31:46 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-13413.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

0xrushi/FocusOnDepth