Occlusion-aware-Amodal-Depth-Estimation-for-Enhancing-3D-Reconstruction-from-a-Single-Image

This repository contains the code of the paper
"Occlusion-aware-Amodal-Depth-Estimation-for-Enhancing-3D-Reconstruction-from-a-Single-Image IEEE Access"

Demo

The following is a demo featuring a bouncing graphic object, utilizing meshes generated by SG3N and the proposed method, along with a Ground Truth (GT) mesh.

Environment

This code was experimented in the following environment

Ubuntu 20.04
Python 3.10
Pytorch 1.10.1
CUDA 11.1
Minkowski Engine 0.5.1, fork
Mask RCNN Benchmark
Nvidia 3090 Ti, 24GB

Installation

a. create conda & install torch

  conda create -n amodal-depth python=3.10
  pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html

b. Install maskrcnn-benchmark

c. Install Forked MinkowskiEngine

git clone https://github.com/xheon/MinkowskiEngine.git
cd MinkowskiEngine
python setup.py install

d. Install Extra python-packages

sudo apt-get install -y openexr
pip install git+https://github.com/jamesbowman/openexrpython.git
pip install git+https://github.com/xheon/PyMarchingCubes.git
pip install SciPy,plyfile,timm, wandb, pandas

Amodal Front-3D

The 3D-FRONT indoor datasets consists of 6,813 furnished apartments.
We used Blenderproc, like SG3N[1], to simulate occlusion images from individual rooms.

Prepare dataset

1.Downlowd zip files weights and data
2.Unzip files
3.Move directories "weights" and "data" to this project
4.GT Mesh for 3D Evaluation

Usage

Train

# 3D-Reconstruction with Amodal Depth
python train_net --output-path {your path}

# 3D-Reconstruction with SG3N
python train_net_base --output-path {your path}

Evaluate

# 3D-Reconstruction with Amodal Depth
# 1.inference eval list 
python test_net_single_iomage_amodal.py --input_list resources/Amodal_front3d/valid.txt --output_path output/amodal/ --config-file configs/amodal_front3d_evaluate.yaml --model weights/SG3N_Amodal.pth

# 2.Configure eval_mesh.py your path then
python eval.mesh.py

# 3D-Reconstruction with SG3N
# 1.inference eval list 
python test_net_single_iomage_base.py --input_list resources/Amodal_front3d/valid.txt --output_path output/amodal/ --config-file configs/amodal_front3d_evaluate.yaml --model weights/SG3N_base.pth

# 2.Configure eval_mesh.py your path then
python eval.mesh.py

demo

# 3D-Reconstruction with Amodal Depth
python test_net_single_iomage_amodal.py --input_list resources/Amodal_front3d/demo.txt --output_path output/amodal/ --config-file configs/amodal_front3d_evaluate.yaml --model weights/SG3N_base.pth

# 3D-Reconstruction with SG3N
python test_net_single_iomage_base.py --input_list resources/Amodal_front3d/demo.txt --output_path output/amodal/ --config-file configs/amodal_front3d_evaluate.yaml --model weights/SG3N_base.pth

Seonguke/Occlusion-aware-Amodal-Depth-Estimation-for-Enhancing-3D-Reconstruction-from-a-Single-Image