Implementation of CVPR 2024 paper "DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior"
DreamControl generates a coarse 3D self-prior and refines texture with control-based distillation, aiming to create 3D content with both high-consistent geometries and high-fidelity textures.
We build our source code based on original threestudio. See installation.md for additional information.
We provide two ways to load shape conditions for our control-based score distillation.
# load an obj file as a condition:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="{your text prompt}" system.geometry_c.shape_init="mesh:{path of the obj file}"# load a threestudio checkpoint as a condition:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="{your text prompt}" system.geometry_c.shape_init="{path of the last checkpoint file}"For example, you can use the template geometry we provide in ./load/rot_shapes or pre-process a new shape with correct orientation (Note that, the orientation of original geoemtries in threestudio's ./load/shapes is different from ours):
# rotate the geometry
cd ./load/rot_shapes
python rot.py ../shapes/nascar.obj
# optimize a NeRF with DreamControl
cd ../..
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="A Tesla Model3 sedan" system.geometry_c.shape_init="mesh:./load/rot_shapes/nascar.obj"Or, you can use a threestudio checkpoint. We provide an example here. You can download and unzip it, and then:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="a corgi" system.geometry_c.shape_init="./a_corgi/ckpts/last.ckpt"- Code of Control-Based Score Distribution.
- Code of 3D Self-Prior Generation.
 |
 |
 |
 |
| "Lionel Messi in a suit, holding the Ballon d'Or" | "Elon musk, using a laptop" | "Batman is riding a moto" | "An astronaut is riding a horse" |
 |
 |
 |
 |
| "A chimpanzee dressed like Henry VIII king of England" | "Michelangelo style statue of dog reading news on a cellphone" | "A blue jay standing on a large basket of rainbow macarons" | "A baby bunny sitting on top of a stack of pancakes" |
 |
 |
 |
 |
| "A model of a house in Tudor style" | "Tower Bridge made out of gingerbread and candy" | "A highly detailed sand castle" | "A pavilion in a Chinese garden" |
 |
 |
 |
 |
| "A gundam" | "Woody from Toy Story" | "Captain America" | "Spider Man" |
 |
 |
 |
 |
| "A wizard" | "A corgi" | "A Tesla Model3 sedan" | "A classic Packard car" |
 |
 |
 |
 |
| "A teddy bear" | "A plate piled high with chocolate chip cookies" | "A plate of fried chicken and waffles with maple syrup on them" | "A vase with pink flowers" |
Our codes are built on threestudio, and ControlLoRA.
@article{huang2023dreamcontrol,
title={Dreamcontrol: Control-based text-to-3d generation with 3d self-prior},
author={Huang, Tianyu and Zeng, Yihan and Zhang, Zhilu and Xu, Wan and Xu, Hang and Xu, Songcen and Lau, Rynson WH and Zuo, Wangmeng},
journal={arXiv preprint arXiv:2312.06439},
year={2023}
}