Text-to-3D with Classifier Score Distillation

Introduction

This repository includes the implementation of applying Classifier Score Distillation (CSD) to a diffusion prior with multi-view capabilities (i.e., MVDream), which effectively addresses the Janus problem.

Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, Xiaojuan Qi

Project Page | Paper

Abstract

Text-to-3D generation has made remarkable progress recently, particularly with methods based on Score Distillation Sampling (SDS) that leverages pre-trained 2D diffusion models. While the usage of classifier-free guidance is well acknowledged to be crucial for successful optimization, it is considered an auxiliary trick rather than the most essential component. In this paper, we re-evaluate the role of classifier-free guidance in score distillation and discover a surprising finding: the guidance alone is enough for effective text-to-3D generation tasks. We name this method Classifier Score Distillation (CSD), which can be interpreted as using an implicit classification model for generation. This new perspective reveals new insights for understanding existing techniques. We validate the effectiveness of CSD across a variety of text-to-3D tasks including shape generation, texture synthesis, and shape editing, achieving results superior to those of state-of-the-art methods.

Art_Deco_lamp._geometric_glass_shade._high-quality_brass_base._intricate_details._8K_resolution._detailed_textures.mp4

Jeep._high-quality._8K_resolution._detailed_textures.mp4

Owl_wearing_a_wizard_robe._full_body._high-quality_textures._8K_resolution._photorealistic.mp4

Jack_Sparrow_from_Pirates_of_the_Caribbean._full_body._high-quality_textures._8K_resolution._photorealistic.mp4

Aloy_from_Horizon_Zero_Dawn._full_body._high-quality_leather_and_metal_armor_textures._8K_resolution._photorealistic.mp4

Motorcycle_racer._full_body._high-quality_racing_suit._8K_resolution._photorealistic.mp4

Installation

The codebase is built upon the MVDream-threestudio framework. Please follow the installation instructions available at the MVDream-threestudio repository for environment setup: MVDream-threestudio Repo.

Running the Code

To run the Classifier Score Distillation (CSD) code, simply execute the provided run.sh script.

Credits

This codebase is built upon the Threestudio and MVDream. Thanks to the authors for their great codebase and contribution to the community.

Citation

Please consider 😬 staring this repository and citing our work if you feel this repository useful.

@article{yu2023text,
  title={Text-to-3d with classifier score distillation},
  author={Yu, Xin and Guo, Yuan-Chen and Li, Yangguang and Liang, Ding and Zhang, Song-Hai and Qi, Xiaojuan},
  journal={arXiv preprint arXiv:2310.19415},
  year={2023}
}

Contact

If you have any questions, you can email me (yuxin27g@gmail.com).

codyreading/Classifier-Score-Distillation