cwchenwang/awesome-3d-diffusion

A collection of papers on diffusion models for 3D generation.

Awesome 3D Diffusion

This repo collects papers that use diffusion models for 3D generation.

🔥🔥🔥 Check out our collection of papers on 4D generation: https://github.com/cwchenwang/awesome-4d-generation

🔥🔥🔥 Please take a look at our survey on diffusion models for 3D Generation, which gives a summary of the papers of this list: https://github.com/cwchenwang/awesome-3d-diffusion/blob/main/survey.pdf

If you consider our paper or list useful, please cite our paper:

@article{wang2024diffusion,
  title={Diffusion Models for 3D Generation: A Survey},
  author={Wang, Chen and Peng, Hao-Yang and Liu, Ying-Tian and Gu, Jiatao and Hu, Shi-Min},
  howpublished = {\url{https://github.com/cwchenwang/awesome-3d-diffusion}},
  year={2024}
}

Note: This list is far from complete, please directly open a pull request if you want to add a paper or modify the information. You don't need to open an issue.

Table of Contents

• 2D Diffusion with Pretraining
  • Text-to-3D Object Generation
  • Compositional or Scene Generation
  • Image-to-3D
  • Human and Animal
  • 3D Editing
  • Texturing
  • Multi-view Diffusion
• 2D Diffusion without Pretraining
  • 3D Objects
  • 3D Scenes
• Diffusion in 3D Space
  • Explicit Representation
  • Implicit Representation
  • Triplane
  • Latent Representation
  • Novel Representations
• Diffusion for Motion
  • Human Motion

2D Diffusion with Pretraining

Text-to-3D Object Generation

• DreamFusion: Text-to-3D using 2D Diffusion, Poole et al., Arxiv 2022
• Magic3D: High-Resolution Text-to-3D Content Creation, Lin et al., Arxiv 2022
• Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation, Wang et al., Arxiv 2022
• Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation, Chen et al., Arxiv 2023
• Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation, Seo et al., Arxiv 2023
• DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Model, Seo et al., Arxiv 2023
• TextMesh: Generation of Realistic 3D Meshes From Text Prompts, Tsalicoglou et al., Arxiv 2023
• Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation, Seo et al., Arxiv 2023
• Text-driven Visual Synthesis with Latent Diffusion Prior, Liao et al., Arxiv 2023
• Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond, Armandpour et al., Arxiv 2023
• HiFA: High-fidelity Text-to-3D with Advanced Diffusion Guidance, Zhu and Zhuang, Arxiv 2023
• ATT3D: Amortized Text-to-3D Object Synthesis, Lorraine et al., Arxiv 2023
• PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation, Li and Bansal, Arxiv 2023
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, Wang et al., NeurIPS 2023.
• DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation, Huang et al., Arxiv 2023
• EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior, Zhao et al., Arxiv 2023
• SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D, Li et al., Arxiv 2023
• DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior, Sun et al., Arxiv 2023
• Text-to-3D with Classifier Score Distillation, Yu et al., Arxiv 2023
• Instant3D: Instant Text-to-3D Generation, Li et al., Arxiv 2023
• HyperFields: Towards Zero-Shot Generation of NeRFs from Text, Babu et al., ICML 2024
• DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation, Yang et al., Arxiv 2023
• Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping, Pan et al., Arxiv 2023
• GaussianDreamer: Fast Generation from Text to 3D Gaussian Splatting with Point Cloud Priors, Yi et al., Arxiv 2023
• LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes, Chung et al., Arxiv 2023
• CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting, Vilesov et al., Arxiv 2023
• LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching, Liang et al., Arxiv 2023
• StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D, Guo et al., Arxiv 2023
• DreamComposer: Controllable 3D Object Generation via Multi-View Conditions, Yang et al., Arxiv 2023
• GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs, Gao et al., Arxiv 2023
• X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation, Ma et al., Arxiv 2023
• HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a Single Image, Wu et al., SIGGRAPH ASIA 2023
• RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D, Qiu et al., Arxiv 2023
• Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors, Ding et al., Arxiv 2023
• Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior, Liu et al., Arxiv 2023
• DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling, Zhou et al., Arxiv 2023
• UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation, Liu et al., Arxiv 2023
• Stable Score Distillation for High-Quality 3D Generation, Tang et al., Arxiv 2023
• DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior, Huang et al., CVPR 2024
• Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation, He et al., Arxiv 2023
• HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generation, Mercier et al., Arxiv 2024
• Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting, Zhang et al., Arxiv 2023
• Retrieval-Augmented Score Distillation for Text-to-3D Generation, Seo et al., Arxiv 2024
• BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis, Jiang and Wang, Arxiv 2024
• DreamReward: Text-to-3D Generation with Human Preference, Ye et al., Arxiv 2024
• DreamFlow: High-Quality Text-to-3D Generation by Approximating Probability Flow, Lee et al., ICLR 2024
• LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis, Xie et al., Arix 2024
• DreamPolisher: Towards High-Quality Text-to-3D Generation via Geometric Diffusion, Lin et al., Arxiv 2024
• VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation, Chen et al., CVPR 2024
• Hash3D: Training-free Acceleration for 3D Generation, Yang and Wang, Arxiv 2024
• MicroDreamer: Zero-shot 3D Generation in ∼20 Seconds by Score-based Iterative Reconstruction, Chen et al., Arxiv 2024
• SketchDream: Sketch-based Text-to-3D Generation and Editing, Liu et al., Arxiv 2024
• Flow Score Distillation for Diverse Text-to-3D, Yan et al., Arxiv 2024
• Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching, Miao et al., Arxiv 2024
• Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication, Chen et al., Arxiv 2024
• DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data, Liu et al., Arxiv 2024
• Text-guided Controllable Mesh Refinement for Interactive 3D Modeling, Chen et al., Arxiv 2024
• PlacidDreamer: Advancing Harmony in Text-to-3D Generation, Huang et al., ACM MM 2024
• JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillation, Jiang et al., ECCV 2024
• DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors, Yan et al., ECCV 2024
• Connecting Consistency Distillation to Score Distillation for Text-to-3D Generation, Li et al., ECCV 2024
• PlacidDreamer: Advancing Harmony in Text-to-3D Generation, Huang et al., Arxiv 2024
• DreamCouple: Exploring High Quality Text-to-3D Generation Via Rectified Flow, Li et al., Arxiv 2024
• ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation, Ma et al., ECCV 2024
• Vista3D: Unravel the 3D Darkside of a Single Image, Shen et al., ECCV 2024
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting, Li et al., Arxiv 2024

Compositional or Scene Generation

• Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models, Höllein et al., Arxiv 2023
• SceneScape: Text-Driven Consistent Scene Generation, Fridman et al., Arxiv 2023
• Compositional 3D Scene Generation using Locally Conditioned Diffusion, Po and Wetzstein, Arxiv 2023
• Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes, Cohen-Bar et al., Arxiv 2023
• CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layout, Lin et al., Arxiv 2023
• Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields, Zhang et al., Arxiv 2023
• Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints, Fang et al., Arxiv 2023
• SceneWiz3D: Towards Text-guided 3D Scene Composition, Zhang et al., Arxiv 2023
• ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priors, Mao et al., Arxiv 2023
• Text2Street: Controllable Text-to-image Generation for Street Views, Su et al., Arxiv 2024
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting, Zhou et al., Arxiv 2023
• Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation, Chen et al., Arxiv 2024
• A Quantitative Evaluation of Score Distillation Sampling Based Text-to-3D, Fei et al., Arxiv 2024
• DreamScene: 3D Gaussian-based Text-to-3D Scene Generation via Formation Pattern Sampling, Li et al., Arxiv 2024
• DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting, Zhou et al., Arxiv 2024
• RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion, Shriram et al., Arxiv 2024
• Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior, Lu et al., Arxiv 2024
• DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling, Yang et al., Arxiv 2024
• Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting, Engstler et al., Arxiv 2024
• REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignment, Han et al., Arxiv 2024
• VividDream: Generating 3D Scene with Ambient Dynamics, Lee et al., Arxiv 2024
• VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation, Chen et al., Arxiv 2024
• Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text, Li et al., Arxiv 2024
• Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches, Xu et al., Arxiv 2024
• Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAE, Yang et al., Arxiv 2024
• HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions, Zhou et al., Arxiv 2024
• COMOGen: A Controllable Text-to-3D Multi-object Generation Framework, Sun et al., Arxiv 2024

Image-to-3D

• NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with $360^{\deg}$ Views, Xu et al., CVPR 2023
• NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors, Deng et al., CVPR 2023
• Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures, Metzer et al., CVPR 2023
• RealFusion: 360{\deg} Reconstruction of Any Object from a Single Image, Melas-Kyriazi et al., Arxiv 2023
• Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior, Tang et al., Arxiv 2023
• Zero-1-to-3: Zero-shot One Image to 3D Object, Liu et al., Arxiv 2023
• DreamBooth3D: Subject-Driven Text-to-3D Generation, Raj et al., Arxiv 2023
• DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Views, Yoo et al., Arxiv 2023
• One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization, Liu et al., Arxiv 2023
• Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors, Qian et al., Arxiv 2023
• 360◦ Reconstruction From a Single Image Using Space Carved Outpainting, Ryu et al., SIGGRAPH ASIA 2023
• Viewpoint Textual Inversion: Unleashing Novel View Synthesis with Pretrained 2D Diffusion Models, Burgess et al., Arxiv 2023
• IPDreamer: Appearance-Controllable 3D Object Generation with Image Prompts, Zeng et al.,
• Customize-It-3D: High-Quality 3D Creation from A Single Image Using Subject-Specific Knowledge Prior, Huang et al., Arxiv 2023
• HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D, Woo et al., Arxiv 2023
• Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting, Zhang et al., Arxiv 2023
• AGG: Amortized Generative 3D Gaussians for Single Image to 3D, Xu et al., Arxiv 2024
• Part123: Part-aware 3D Reconstruction from a Single-view Image, Liu et al., SIGGRAPH Conference 2024
• GECO: Generation Image-to-3D within a Second, Wang et al., Arxiv 2024
• Fourier123: One Image to High-Quality 3D Object Generation with Hybrid Fourier Score Distillation, Yang et al., Arxiv 2024
• Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image, Wu et al., Arxiv 2024

Human and Animal

• ScoreHMR: Score-Guided Diffusion for 3D Human Recovery, Stathopoulos et al., CVPR 2024
• Generative Proxemics: A Prior for 3D Social Interaction from Images, Müller at al, CVPR 2024
• DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance, Zhang et al., Arxiv 2023
• AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control, Jiang et al., ICCV 2023
• DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models, Cao et al., Arxiv 2023
• DreamWaltz: Make a Scene with Complex 3D Animatable Avatars, Huang et al., Arxiv 2023
• ZeroAvatar: Zero-shot 3D Avatar Generation from a Single Image, Weng et al., Arxiv 2023
• AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation, Zeng et al., Arxiv 2023
• Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion Jakab et al., Arxiv 2023
• Anything 3D: Towards Single-view Anything Reconstruction in the Wild, Shen et al., Arxiv 2023
• ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections, Yao et al., Arxiv 2023
• TADA! Text to Animatable Digital Avatars, Liao et al., Arxiv 2023
• Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips, Ye et al., ICCV 2023
• Text-Guided Generation and Editing of Compositional 3D Avatars, Zhang et al., Arxiv 2023
• HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting, Liu et al., Arxiv 2023
• AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text, Zhang et al., Arxiv 2023
• Disentangled Clothed Avatar Generation from Text Descriptions, Wang et al., Arxiv 2023
• SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance, Xu et al., Arxiv 2023
• GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning, Yuan et al., Arxiv 2023
• Make-A-Character: High Quality Text-to-3D Character Generation within Minutes, Ren et al., Arxiv 2023
• Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation, Chen et al., Arxiv 2024

3D Editing

• SKED: Sketch-guided Text-based 3D Editing, Mikaeili et al., Arxiv 2023
• Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions, Haque et al., ICCV 2023
• Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversion, Kamata et al., Arxiv 2023
• Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model, Yu et al., Arxiv 2023
• Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D Diffusion-based Editor, Shao et al., Arxiv 2023
• RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models, Zhou et al., Arxiv 2023
• DreamEditor: Text-Driven 3D Scene Editing with Neural Fields, Zhuang et al., SIGRRAPH ASIA 2023
• Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates, Shum et al., Arxiv 2023
• ProteusNeRF: Fast Lightweight NeRF Editing using 3D-Aware Image Context, Wang et al., Arxiv 2023
• ED-NeRF: Efficient Text-Guided Editing of 3D Scene using Latent Space NeRF, Park et al., Arxiv 2023
• 3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation, Decatur et al., Arxiv 2023
• GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting, Chen et al., Arxiv 2023
• Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion, Prabhu et al., Arxiv 2023
• NeRFiller: Completing Scenes via Generative 3D Inpainting, Weber et al., Arxiv 2023
• SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds, Chen et al., Arxiv 2023
• LatentEditor: Text Driven Local Editing of 3D Scenes, Khalid et al., Arxiv 2023
• Free-Editor: Zero-shot Text-driven 3D Scene Editing, Karim et al., Arxiv 2023
• SIGNeRF: Scene Integrated Generation for Neural Radiance Fields, Dihlmann et al., Arxiv 2023
• Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Models, Song et al., Arxiv 2024
• ReplaceAnything3D: Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields, Bartrum et al., Arxiv 2024
• GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing, Wu et al., Arxiv 2024
• View-Consistent 3D Editing with Gaussian Splatting, Wang et al., Arxiv 2024
• Interactive3D: Create What You Want by Interactive 3D Generation, Dong et al., Arxiv 2024
• DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing, Chen et al., Arxiv 2024
• DATENeRF: Depth-Aware Text-based Editing of NeRFs, Rojas et al., Arxiv 2024

Texturing

• TEXTure: Text-Guided Texturing of 3D Shapes, Richardson et al., Arxiv 2023
• Text2Tex: Text-driven Texture Synthesis via Diffusion Models, Chen et al., Arxiv 2023
• EucliDreamer: Fast and High-Quality Texturing for 3D Models with Stable Diffusion Depth, Le et al., Arxiv 2023
• Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models, Zeng et al., Arxiv 2023
• TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion, Yeh et al., Arxiv 2024
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes, Zheng et al., Arxiv 2024
• DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models, Zhang et al., Arxiv 2024
• TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling, Huo et al., Arxiv 2024

Multi-view Diffusion

• MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion, Tang et al., Arxiv 2023
• MVDream: Multi-view Diffusion for 3D Generation, Shi et al., Arxiv 2023
• SyncDreamer: Generating Multiview-consistent Images from a Single-view Image, Liu et al., Arxiv 2023
• Consistent123: Improve Consistency for One Image to 3D Object Synthesis, Weng et al., Arxiv 2023
• Wonder3D: Single Image to 3D using Cross-Domain Diffusion, Long et al., Arxiv 2023
• Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model, Shi et al., Arxiv 2023
• TOSS:High-quality Text-guided Novel View Synthesis from a Single Image, Shi et al., Arxiv 2023
• Text-Guided Texturing by Synchronized Multi-View Diffusion, Liu et al., Arxiv 2023
• Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion, Lu et al., Arxiv 2023
• ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models, Kwak et al., Arxiv 2023
• EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion, Huang et al., CVPR 2024
• BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion, Yu et al., Arxiv 2024
• EscherNet: A Generative Model for Scalable View Synthesis, Kong et al., Arxiv 2024
• LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation, Tang et al., Arxiv 2024
• SPAD : Spatially Aware Multiview Diffusers, Kant et al., Arxiv 2024
• IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generation, Melas-Kyriazi et al., Arxiv 2024
• MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction, Tang et al., Arxiv 2024
• CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model, Wang et al., Arxiv 2024
• V3D: Video Diffusion Models are Effective 3D Generators, Chen et al., Arxiv 2024
• Envision3D: One Image to 3D with Anchor Views Interpolation, Pang et al., Arxiv 2024
• Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation, Liu et al., Arxiv 2024
• Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting, Li et al., Arxiv 2024
• FDGaussian: Fast Gaussian Splatting from Single Image via Geometric-aware Diffusion Model, Feng et al., Arxiv 2024
• Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding, Liu et al., Arxiv 2024
• SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, Vikram et al., Arxiv 2024
• Generic 3D Diffusion Adapter Using Controlled Multi-View Editing, Chen et al., Arxiv 2024
• VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models, Han et al., Arxiv 2024
• Garment3DGen: 3D Garment Stylization and Texture Generation, Zhang et al., Arxiv 2024
• MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation, Hu et al., CVPR 2024
• Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion, Yang et al., Arxiv 2024
• InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models, Xu et al., Arxiv 2024
• Magic-Boost: Boost 3D Generation with Mutli-View Conditioned Diffusion, Yang et al., Arxiv 2024
• Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model, Li et al., Arxiv 2024
• Multi-view Image Prompted Multi-view Diffusion for Improved 3D Generation, Kim et al., Arxiv 2024
• MVDiff: Scalable and Flexible Multi-View Diffusion for 3D Object Reconstruction from Single-View, Bourigault et al., Arxiv 2024
• CAT3D: Create Anything in 3D with Multi-View Diffusion Models, Gao et al., Arxiv 2024
• CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refiner, Li et al., Arxiv 2024
• Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion, Wen et al., Arxiv 2024
• Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle, Tang et al., Arxiv 2024

2D Diffusion without Pretraining

3D Objects

• Novel View Synthesis with Diffusion Models, Watson et al., ICLR 2023
• Generative Novel View Synthesis with 3D-Aware Diffusion Models, Chan et al., Arxiv 2023
• NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion, Gu et al., ICML 2023
• 3DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models, Li et al., Arxiv 2022
• SparseFusSparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction, Zhou and Tulsiani, CVPR 2023
• HoloDiffusion: Training a 3D Diffusion Model using 2D Images, Karnewar et al., CVPR 2023
• Renderdiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation, Anciukevičius et al., CVPR 2023
• Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision, Tewari et al., Arxiv 2023
• 3D-aware Image Generation using 2D Diffusion Models, Xiang et al., Arxiv 2023
• Viewset Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data, Szymanowicz et al., Arxiv 2023
• HOLOFUSION: Towards Photo-realistic 3D Generative Modeling, Karnewar et al., Arxiv 2023
• ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image, Sargent et al., Arxiv 2023
• Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model, Li et al., Arxiv 2023
• DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model, Xu et al., Arxiv 2023
• LRM: Large Reconstruction Model for Single Image to 3D, Hong et al., Arxiv 2023
• WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space, Schwarz et al., Arxiv 2023
• ViewFusion: Learning Composable Diffusion Models for Novel View Synthesis, Spiegl et al., Arxiv 2024

3D Scenes

• Consistent View Synthesis with Pose-Guided Diffusion Models, Tseng et al., CVPR 2023
• Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models, Yu et al., Arxiv 2023
• DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models, Cai et al., Arxiv 2023
• SemCity: Semantic Scene Generation with Triplane Diffusion, Lee et al., CVPR 2024

Diffusion in 3D Space

3D Gaussians

• GVGEN: Text-to-3D Generation with Volumetric Representation, He et al., Arxiv 2024
• GaussianCube: Structuring Gaussian Splatting using Optimal Transport for 3D Generative Modeling, Zhang et al., Arxiv 2024
• Atlas Gaussians Diffusion for 3D Generation with Infinite Number of Points, Yang et al., Arxiv 2024

Point Cloud, Meshs, Volumes

• Diffusion Probabilistic Models for 3D Point Cloud Generation, Luo et al., CVPR 2021
• 3d shape generation and completion through point-voxel diffusion, Zhou et al., Arxiv 2021
• A Diffusion-ReFinement Model for Sketch-to-Point Modeling, Kong et al., ACCV 2022
• Controllable Mesh Generation Through Sparse Latent Point Diffusion Models, Lyu et al., CVPR 2023
• Point-E: A System for Generating 3D Point Clouds from Complex Prompts, Nichol et al., ICML 2023
• DiffFacto: Controllable Part-Based 3D Point Cloud Generation with Cross Diffusion, Nakayama et al., Arxiv 2023
• Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation, Wu et al., ICCV 2023
• DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation, Mo et al., Arxiv 2023
• MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers, Siddiqui et al., CVPR 2024
• ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model, Yin et al., Arxiv 2023
• VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder, Tang et al., Arxiv 2023
• SPiC·E: Structural Priors in 3D Diffusion Models using Cross-Entity Attention, Sella et al., Arxiv 2023
• PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models, Alliegro et al., Arxiv 2023

Implicit Representation

• Learning A Diffusion Prior For Nerfs, Yang et al., ICLRW 2023
• Tetrahedral Diffusion Models for 3D Shape Generation, Nikolai and Torben et al., Arxiv 2022
• MeshDiffusion: Score-based Generative 3D Mesh Modeling, Liu et al., ICLR 2023
• Neural Wavelet-domain Diffusion for 3D Shape Generation, Hui et al., SIGGRAPH Asia 2022
• Neural Wavelet-domain Diffusion for 3D Shape Generation, Inversion, and Manipulation, Hu and Hui et al., Arxiv 2023
• DiffRF: Rendering-Guided 3D Radiance Field Diffusion, Muller et al., CVPR 2023
• Locally Attentional SDF Diffusion for Controllable 3D Shape Generation, Zheng et al., SIGGRAPH 2023
• HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion, Erkoç et al., ICCV 2023
• DiffComplete: Diffusion-based Generative 3D Shape Completion, Chu et al., Arxiv 2023
• DiffRoom: Diffusion-based High-Quality 3D Room Reconstruction and Generation, Ju et al., Arxiv 2023
• Surf-D: High-Quality Surface Generation for Arbitrary Topologies using Diffusion Models, Yu et al., Arxiv 2023

Triplane

• 3D Neural Field Generation using Triplane Diffusion, Shue et al., Arxiv 2022
• DiffusionSDF: Conditional Generative Modeling of Signed Distance Functions, Chou et al., Arxiv 2022
• Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion, Wang et al., CVPR 2023
• 3DGen: Triplane Latent Diffusion for Textured Mesh Generation, Gupta et al., Arxiv 2023
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction, Chen et al., Arxiv 2023
• Learning Controllable 3D Diffusion Models from Single-view Images, Gu et al., Arxiv 2023
• Compress3D: a Compressed Latent Space for 3D Generation from a Single Image, Zhang et al., Arxiv 2024
• Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion, Li et al., Arxiv 2024
• Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer, Wu et al., Arxiv 2024

Latent Representation

• GAUDI: A Neural Architect for Immersive 3D Scene Generation, Bautista et al., NeurIPS 2022
• LION: Latent Point Diffusion Models for 3D Shape Generation, Zeng et al., NeurIPS 2022
• Diffusion-SDF: Text-to-Shape via Voxelized Diffusion, Li et al., CVPR 2023
• 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models, Nam et al., Arxiv 2022
• 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models, Zhang et al., SIGGRAPH 2023
• Shap-E: Generating Conditional 3D Implicit Functions, Jun et al., Arxiv 2023
• StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation, Zhang et al., Arxiv 2023
• AutoDecoding Latent 3D Diffusion Models, Ntavelis et al., Arxiv 2023
• XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies, Ren et al., CVPR 2024
• LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation, Lan et al., ECCV 2024
• GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation, Lan et al., Arxiv 2024
• Structured 3D Latents for Scalable and Versatile 3D Generation, Xiang et al., Arxiv 2024

Novel Representations

• X-Ray: A Sequential 3D Representation for Generation, Hu et al., Arxiv 2024
• Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation, Elizarov et al., Arxiv 2024
• 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion, Chen et al., Arxiv 2024
• An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion, Yan et al., Arxiv 2024

Diffusion for Motion

Human Motion

• SMooDi: Stylized Motion Diffusion Model, Lei et al., ECCV 2024
• Single Motion Diffusion, Raab et al., ICLR 2024
• OmniControl: Control Any Joint at Any Time for Human Motion Generation, Xie et al., ICLR 2024
• Human Motion Diffusion as a Generative Prior, Shafir et al., ICLR 2024
• MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation, Hoang et al., AAAI 2024
• DNO: Optimizing Diffusion Noise Can Serve As Universal Motion Priors, Karunratanakul et al., Arxiv 2023
• RoHM: Robust Human Motion Reconstruction via Diffusion, Zhang et al., Arxiv 2023
• EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Human Motion Generation, Zhou et al., Arxiv 2023
• DiffusionPhase: Motion Diffusion in Frequency Domain, Wan et al., Arxiv 2023
• InterControl: Generate Human Motion Interactions by Controlling Every Joint, Wang et al., Arxiv 2023
• AAMDM: Accelerated Auto-regressive Motion Diffusion Model, Li et al., Arxiv 2023
• ReMoS: Reactive 3D Motion Synthesis for Two-Person Interactions, Ghosh et al., Arxiv 2023
• HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models, Peng et al., Arxiv 2023
• Controllable Motion Diffusion Model, Shi et al., Arxiv 2023
• MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion, Kapon et al., Arxiv 2023
• CG-HOI: Contact-Guided 3D Human-Object Interaction Generation, Diller et al., Arxiv 2023
• A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis, Zhou et al., Arxiv 2023
• Controllable Motion Synthesis and Reconstruction with Autoregressive Diffusion Models, Yin et al., Arxiv 2023
• Guided Motion Diffusion for Controllable Human Motion Synthesis, Karunratanakul et al., ICCV 2023
• Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models, Pi et al., ICCV 2023
• PhysDiff: Physics-Guided Human Motion Diffusion Model, Yuan et al., ICCV 2023
• Priority-Centric Human Motion Generation in Discrete Latent Space, Kong et al., ICCV 2023
• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model, Zhang et al., ICCV 2023
• Flame: Free-form language-based motion synthesis & editing, Kim et al., AAAI 2023
• Object Motion Guided Human Motion Synthesis, Li et al., SIG ASIA 2023, TOG 2023
• Controllable Group Choreography using Contrastive Diffusion, Le et al., TOG 2023
• Listen, denoise, action! Audio-driven motion synthesis with diffusion models, Alexanderson et al., SIG 2023, TOG 2023
• GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents Ao et al., SIG 2023, TOG 2023
• Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation, Zhu et al., CVPR 2023
• MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis, Dabral et al., CVPR 2023
• Executing your Commands via Motion Diffusion in Latent Space, Jiang et al., CVPR 2023
• MDM: Human Motion Diffusion Model, Tevet et al., ICLR 2023
• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model, Zhang et al., Arxiv 2022