/articulated-object-nerf

Experimental repo for modelling Articulated Object Neural Radiance Field

Primary LanguagePython

Articulated Object Neural Radiance Field

This repository is the experimental implementation of articulated object NeRF training and rendering from some of the ideas mentioned in a few papers including CLA-NeRF and NASAM.

Goal of this repo: To implement a simple and intuitive pipeline for training and rendering Neural Radiance Fields for articulated objects given 1. Multi-view consistent as well as articulation-aware input images of specific objects (Overfitting) and 2. (In pipeline) Multi-view consistent and articulation-aware input images of diverse instances of objects within a category to predict NeRFs for new instances using few images (Generalization)

Overall, this repo can be used as stand-alone project for creating your articulated object NeRF models and building other cool things on top of it. The project is built on top of amazing NeRF implementation of nerf_pl and nerf-factory. Kudos to the authors for creating wonderful repos!

If you find this repository useful, please consider citing and STARing🌟 the repo. Feel free to share the implementation with others!

Contents

🍀 Overview

Experimental Repo for Modelling Neural Radiance Field for Articulated Objects. Currently supported Experiments:

  • Sapien Dataset (Single Instance Overfitting)
  • Sapien Dataset (Single Instance Articulated Overfitting)
  • Sapien Dataset (Single Instance Auto-Encoder Articulated NeRF)

Future (In Pipeline):

  • Sapien Dataset (Single Instance Auto-Decoder Articulated NeRF)
  • Sapien Dataset (Generalization experiments i.e. few-shot Articulated NeRF)

💻 Installation

  1. Hardware
  • OS: Ubuntu 18.04
  • NVIDIA GPU with CUDA>=10.2 (tested with 1 RTX2080Ti)
  1. Software
  • Clone this repo by git clone --recursive https://github.com/zubair-irshad/articulated-object-nerf
  • Python>=3.7 (installation via anaconda is recommended, use conda create -n ao-nerf python=3.7 to create a conda environment and activate it by conda activate aon)
  • Python libraries
    • Install core requirements by pip install -r requirements.txt

📊 Dataset

  1. Download pre-generated SAPIEN Dataset: Download link1 Download link2

  2. Generate Sapien Dataset

    • Data generation for NeRF training
      1. Modify the config file in ./config/data_gen_*.json
      2. (optional) If you want to save the pose for image rendering, add key-value "save_render_pose_path": path/to/save in the json file. If you want to generate images using this saved pose file, add key-value "render_pose_path":path/to/files.
      3. Once you have modified your config file, just run python datagen/data_gen.py --config config/your_datagen_config.json.
      4. Data will be saved in output_dir/{train, val, test}

    scissor is the output_dir, and the folder tree looks like this

    scissor
├── test
├── train
│   ├── depth
│   ├── rgb
│   └── transforms.json
└── val

    The generated images looks like this Image

    • Data generation for Articulation training
      • Coming soon!

✨ Training and Inference

  1. Setup wandb for logging
  • run wandb login in command line and enter your wandb key to setup
  1. Training
  • Single Instance Overfitting training

Note that the --root_dir shall be the same as the --output_dir specified in data generation.

python run.py --dataset_name sapien --root_dir /experiments/zubair/datasets/sapien_single_scene --exp_name sapien_single_scene_vanilla --exp_type vanilla --img_wh 640 480 --white_back --batch_size 1 --num_gpus 8  --num_epochs 100

Alternatively

You can wirte all those args in the json config file as key-value pair, check config/nerf_training.json for example. Once you have modified your config file, run: python run.py --config config/nerf_training.json.

Same for evaluation(test).

  • Single Instance Articulated Overfitting
CUDA_VISIBLE_DEVICES=0,2,3,4,5,6,7 python run.py --dataset_name sapien_multi --root_dir /experiments/zubair/datasets/sapien_single_scene_art --exp_name sapien_single_scene_articulated --exp_type vanilla_autodecoder --img_wh 320 240 --white_back --batch_size 1 --num_gpus 7
  1. Evaluation
  • Single Scene Overfitting:
CUDA_VISIBLE_DEVICES=0 python run.py --dataset_name sapien --root_dir /experiments/zubair/datasets/sapien_single_scene --exp_name sapien_single_scene_vanilla --exp_type vanilla --img_wh 640 480 --white_back --batch_size 1 --num_gpus 1 --run_eval --render_name sapien_test_highres
  • Single Insatnce Articulation Overfitting
CUDA_VISIBLE_DEVICES=0 python /home/ubuntu/zubair/articulated-object-nerf/run.py --dataset_name sapien_multi --root_dir /experiments/zubair/datasets/sapien_single_scene_art --exp_name sapien_single_scene_articulated --exp_type vanilla_autodecoder --img_wh 320 240 --white_back --batch_size 1 --N_max_objs 1 --run_eval --render_name single_image_train_degs_interpolation2

📝 Citation

If you find this repository useful, please consider citing this implementation as well as the two original papers:

@misc{irshad2023articulatednerf,
    title = {Articulated Object NeRF - A simple and intuitive implementation of creating articulated object neural radiance fields},
    author = {Muhammad Zubair Irshad},
    journal = {GitHub repository},
    url = {https://github.com/zubair-irshad/articulated-object-nerf},
    year = {2023},
}

@inproceedings{tseng2022clanerf,
      title={CLA-NeRF: Category-Level Articulated Neural Radiance Field}, 
      author={Wei-Cheng Tseng and Hung-Ju Liao and Yen-Chen Lin and Min Sun},
      booktitle={ICRA},
      year={2022},
}

@inproceedings{wei2022nasam,
    title = {Self-supervised Neural Articulated Shape and Appearance Models},
    author = {Fangyin Wei and Rohan Chabra and Lingni Ma and Christoph
              Lassner and Michael Zollhoefer and Szymon Rusinkiewicz and Chris
              Sweeney and Richard Newcombe and Mira Slavcheva},
    booktitle = {Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2022}
}