DreamFusionAcc

Overview

This is a minimal PyTorch implementation of DreamFusion and its variant Magic3D, where we utilize Instant-NGP as the neural renderer and Stable Diffusion/DeepFloyd IF as the guidance.

It takes ~30min to train with Stable Diffusion and ~40min with DeepFloyd IF on a single 3090.

⚠️ This repo has not yet been well-optimized, e.g., memory usage and lack of refinement stage. However, it is still a good reference for its easy-to-follow and lightweight implementation. Please use stable-dreamfusion or threestudio for better quality 3D generation and mesh export.

Installation

git clone https://github.com/chinhsuanwu/dreamfusionacc.git
cd dreamfusionacc
pip install -r requirements.txt

Dependencies (click to expand)

Dependencies

torch
tinycudann
nerfacc
numpy
imageio
einops
diffusers
trainsformers

You may install the pre-built wheels for NerfAcc. Please check out here. This repo is built upon torch 1.13.0 + cu117.

Howtos

To train

python train.py --config config/peacock.yaml

You can find all controllable settings in the yaml file. After the training is done, run

python test.py --config config/peacock.yaml

to render 360˚ visualizations.

Citation

@article{poole2022dreamfusion,
  author = {Poole, Ben and Jain, Ajay and Barron, Jonathan T. and Mildenhall, Ben},
  title = {DreamFusion: Text-to-3D using 2D Diffusion},
  journal = {arXiv},
  year = {2022},
}

@inproceedings{lin2023magic3d,
  title={Magic3D: High-Resolution Text-to-3D Content Creation},
  author={Lin, Chen-Hsuan and Gao, Jun and Tang, Luming and Takikawa, Towaki and Zeng, Xiaohui and Huang, Xun and Kreis, Karsten and Fidler, Sanja and Liu, Ming-Yu and Lin, Tsung-Yi},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2023}
}

Credits

This implementation is heavily based on NerfAcc and stable-dreamfusion. Kudos to the authors for their amazing work!