/awesome-text-to-image-studies

A collection of awesome text-to-image generation studies.

Primary LanguageTeXMIT LicenseMIT

A Collection of Text-to-Image Generation Studies

This GitHub repository summarizes papers and resources related to the text-to-image generation task.

If you have any suggestions about this repository, please feel free to start a new issue or pull requests.

🔥 News

Contents

To-Do Lists

  • Published Papers on Conferences
    • Update CVPR 2024 Papers
    • Update AAAI 2024 Papers
      • Update ⚠️ Papers and References
      • Update arXiv References into CVPR and AAAI Versions
    • Update ICLR 2024 Papers
    • Update NeurIPS 2024 Papers
  • Create A List with only Diffusion Model-based Papers
  • Regular Maintenance of Preprint arXiv Papers and Missed Papers

<🎯Back to Top>

Products

Name Year Website Specialties
Stable Diffusion 3 2024 link Diffusion Transformer-based Stable Diffusion
Stable Video 2024 link High-quality high-resolution images
DALL-E 3 2023 link Collaborate with ChatGPT
Ideogram 2023 link Text images
Playground 2023 link Athestic images
HiDream.ai 2023 link -
Dashtoon 2023 link Text-to-Comic Generation
Midjourney 2022 link Powerful close-sourced generation tool

<🎯Back to Top>

Papers

Survey Papers

  • Text-to-Image Generation
    • Year 2024
      • ACM Computing Surveys
        • Diffusion Models: A Comprehensive Survey of Methods and Applications [Paper]
    • Year 2023
      • TPAMI
      • arXiv
        • Text-to-image Diffusion Models in Generative AI: A Survey [Paper]
        • State of the Art on Diffusion Models for Visual Computing [Paper]
    • Year 2022
      • arXiv
        • Efficient Diffusion Models for Vision: A Survey [Paper]
  • Conditional Text-to-Image Generation
    • Year 2024
      • arXiv
        • Controllable Generation with Text-to-Image Diffusion Models: A Survey [Paper]
  • Text-Guided Image Editing
    • Year 2024
      • arXiv

<🎯Back to Top>

Text-to-Image Generation

  • Year 2024
    • CVPR
      • DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models [Paper] [Code]
      • InstanceDiffusion: Instance-level Control for Image Generation [Paper] [Code] [Project]
      • ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations [Paper] [Code] [Project] [Demo]
      • Instruct-Imagen: Image Generation with Multi-modal Instruction [Paper]
      • Learning Continuous 3D Words for Text-to-Image Generation [Paper] [Code]
      • HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances [Paper]
      • Rich Human Feedback for Text-to-Image Generation [Paper]
      • MarkovGen: Structured Prediction for Efficient Text-to-Image Generation [Paper]
      • Customization Assistant for Text-to-image Generation [Paper]
      • ADI: Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation [Paper] [Project]
      • UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs [Paper]
      • Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation [Paper]
      • Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting [Paper] [Code]
      • CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation [Paper] [Code] [Project] [Demo]
      • ⚠️ Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder [Paper]
      • ⚠️ On the Scalability of Diffusion-based Text-to-Image Generation [Paper]
      • ⚠️ MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation [Paper]
      • ⚠️ Discriminative Probing and Tuning for Text-to-Image Generation [Paper]
      • ⚠️ Learning Multi-dimensional Human Preference for Text-to-Image Generation [Paper]
      • ⚠️ Towards Effective Usage of Human-Centric Priors in Diffusion Models for Text-based Human Image Generation [Paper]
      • ⚠️ Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning [Paper]
      • ⚠️ Adversarial Text to Continuous Image Generation [Paper]
      • ⚠️ Dynamic Prompt Optimizing for Text-to-Image Generation [Paper]
    • ICLR
      • Patched Denoising Diffusion Models For High-Resolution Image Synthesis [Paper] [Code]
      • Relay Diffusion: Unifying diffusion process across resolutions for image synthesis [Paper] [Code]
      • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis [Paper] [Code]
      • Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis [Paper] [Code]
      • PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis [Paper] [Code] [Project] [Demo]
    • AAAI
      • Semantic-aware Data Augmentation for Text-to-image Synthesis [Paper]
      • ⚠️ Text-to-Image Generation for Abstract Concepts [Paper]
    • arXiv
      • Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [Paper]
      • RPG: Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs [Paper] [Code]
      • Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation [Paper] [Code]
      • ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models [Paper] [Code] [Project]
      • DiS: Scalable Diffusion Models with State Space Backbone [Paper] [Code]
      • InstantID: Zero-shot Identity-Preserving Generation in Seconds [Paper] [Code] [Project] [Demo]
      • PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models [Paper] [Code]
      • PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation [Paper] [Code] [Project]
      • CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion [Paper] [Code]
      • ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment [Paper] [Code] [Project]
      • Text2Street: Controllable Text-to-image Generation for Street Views [Paper]
    • Others

<🎯Back to Top>

  • Year 2023
    • CVPR
      • GigaGAN: Scaling Up GANs for Text-to-Image Synthesis [Paper] [Reproduced Code] [Project] [Video]
      • ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model With Knowledge-Enhanced Mixture-of-Denoising-Experts [Paper]
      • Shifted Diffusion for Text-to-image Generation [Paper] [Code]
      • GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis [Paper] [Code]
      • Specialist Diffusion: Plug-and-Play Sample-Efficient Fine-Tuning of Text-to-Image Diffusion Models to Learn Any Unseen Style [Paper] [Code]
      • Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation [Paper]
      • RIATIG: Reliable and Imperceptible Adversarial Text-to-Image Generation with Natural Prompts [Paper] [Code]
    • NeurIPS
      • ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation [Paper] [Code]
      • RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths [Paper] [Project]
    • ICLR
      • Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis [Paper] [Code]
    • ICML
    • ACM MM
      • SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with Large Language Models [Paper] [Code]
      • ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors [Paper]
    • SIGGRAPH
    • arXiv
      • P+: Extended Textual Conditioning in Text-to-Image Generation [Paper]
      • SDXL-Turbo: Adversarial Diffusion Distillation [Paper] [Code]
      • Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models [Paper] [Code]
      • StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation [Paper] [Project]
    • Others
      • DALL-E 3: Improving Image Generation with Better Captions [Paper]

<🎯Back to Top>

  • Year 2022
    • CVPR
      • 🔥 Stable Diffusion: High-Resolution Image Synthesis With Latent Diffusion Models [Paper] [Code] [Project]
      • Vector Quantized Diffusion Model for Text-to-Image Synthesis [Paper] [Code]
      • DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis [Paper] [Code]
      • LAFITE: Towards Language-Free Training for Text-to-Image Generation [Paper] [Code]
      • Text-to-Image Synthesis based on Object-Guided Joint-Decoding Transformer [Paper]
      • StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis [Paper] [Code]
    • ECCV
      • Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors [Paper] [Code] [Demo]
      • Trace Controlled Text to Image Generation [Paper]
      • Improved Masked Image Generation with Token-Critic [Paper]
      • VQGAN-CLIP: Open Domain Image Generation and Manipulation Using Natural Language [Paper] [Code]
      • TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation [Paper] [Code]
      • StoryDALL-E: Adapting Pretrained Text-to-image Transformers for Story Continuation [Paper] [Code] [Demo]
    • NeurIPS
    • ACM MM
      • Adma-GAN: Attribute-Driven Memory Augmented GANs for Text-to-Image Generation [Paper] [Code]
      • Background Layout Generation and Object Knowledge Transfer for Text-to-Image Generation [Paper]
      • DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation [Paper]
      • AtHom: Two Divergent Attentions Stimulated By Homomorphic Training in Text-to-Image Synthesis [Paper]
    • arXiv
      • DALLE-2: Hierarchical Text-Conditional Image Generation with CLIP Latents [Paper]
      • PITI: Pretraining is All You Need for Image-to-Image Translation [Paper] [Code]

<🎯Back to Top>

  • Year 2021
    • ICCV
      • DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis [Paper] [Code]
    • NeurIPS
      • CogView: Mastering Text-to-Image Generation via Transformers [Paper] [Code] [Demo]
      • UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis [Paper]
    • ICML
    • ACM MM
      • Cycle-Consistent Inverse GAN for Text-to-Image Synthesis [Paper]
      • R-GAN: Exploring Human-like Way for Reasonable Text-to-Image Synthesis via Generative Adversarial Networks [Paper]

<🎯Back to Top>

  • Year 2020
    • ACM MM
      • Text-to-Image Synthesis via Aesthetic Layout [Paper]

<🎯Back to Top>

Conditional Text-to-Image Generation

  • Year 2024
    • CVPR
      • PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis [Paper]
      • One-Shot Structure-Aware Stylized Image Synthesis [Paper]
      • Grounded Text-to-Image Synthesis with Attention Refocusing [Paper] [[Project]]
      • Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [Paper] [Code]
      • ⚠️ Zero-Painter: Training-Free Layout Control for Text-to-Image Synthesis [Paper]
    • ICLR
      • Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models [Paper] [Code]
    • WACV
    • AAAI
      • SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-image Generation [Paper]
      • Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models [Paper] [Code]

<🎯Back to Top>

  • Year 2023
    • CVPR
    • ICCV
      • ControlNet: Adding Conditional Control to Text-to-Image Diffusion Models [Paper] [Code]
      • SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis [Paper] [Code]
    • ICML
    • SIGGRAPH
    • NeurIPS
    • WACV
      • More Control for Free! Image Synthesis with Semantic Diffusion Guidance [Paper]
    • ACM MM
    • LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [Paper]
    • arXiv
      • T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models [Paper] [Code] [Demo]
      • BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing [Paper] [Code]

<🎯Back to Top>

Personalized Text-to-Image Generation

  • Year 2024
    • CVPR
      • Cross Initialization for Personalized Text-to-Image Generation [Paper]
      • When StyleGAN Meets Stable Diffusion: a W+ Adapter for Personalized Image Generation [Paper] [Code] [Project]
      • Style Aligned Image Generation via Shared Attention [Paper] [Code] [Project]
      • InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning [Paper] [Project]
      • High Fidelity Person-centric Subject-to-Image Synthesis [Paper]
      • RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization [Paper] [Project]
      • ⚠️ FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition [Paper]
      • ⚠️ JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [Paper]
      • ⚠️ Countering Personalized Text-to-Image Generation with Influence Watermarks [Paper]
      • ⚠️ Personalized Residuals for Concept-Driven Text-to-Image Generation [Paper]
      • ⚠️ Improving Subject-Driven Image Synthesis with Context-Agnostic Guidance [Paper]
    • AAAI
      • Decoupled Textual Embeddings for Customized Image Generation [Paper]
  • Year 2023
    • CVPR
    • ICCV
      • ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation [Paper] [Code]
    • ICLR
      • Textual Inversion: An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion [Paper] [Code] [Project]
    • SIGGRAPH
      • Break-A-Scene: Extracting Multiple Concepts from a Single Image [Paper] [Code]
      • Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models [Paper] [Project]
    • arXiv
      • DreamTuner: Single Image is Enough for Subject-Driven Generation [Paper] [Project]
      • PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding [Paper] [Code]

<🎯Back to Top>

Text-Guided Image Editing

  • Year 2024
    • CVPR
      • InfEdit: Inversion-Free Image Editing with Natural Language [Paper] [Code] [Project]
      • Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing [Paper]
      • Doubly Abductive Counterfactual Inference for Text-based Image Editing [Paper] [Code]
      • Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation [Paper] [Code]
      • Contrastive Denoising Score for Text-guided Latent Diffusion Image Editing [Paper]
      • DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [Paper] [Code]
      • DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing [Paper]
      • FreeDrag: Feature Dragging for Reliable Point-based Image Editing [Paper] [Code]
      • Text-Driven Image Editing via Learnable Regions [Paper] [Code] [Project] [Video]
      • LEDITS++: Limitless Image Editing using Text-to-Image Models [Paper] [Code] [Project] [Demo]
      • SmartEdit: Exploring Complex Instruction-based Image Editing with Large Language Models [Paper] [Code] [Project]
      • Edit One for All: Interactive Batch Image Editing [Paper] [Code] [Project]
      • ⚠️ TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing [Paper]
      • ⚠️ Person in Place: Generating Associative Skeleton-Guidance Maps for Human-Object Interaction Image Editing [Paper]
      • ⚠️ Referring Image Editing: Object-level Image Editing via Referring Expressions [Paper]
      • ⚠️ The Devil is in the Details: StyleFeatureEditor for Detail-Rich StyleGAN Inversion and High Quality Image Editing [Paper]
      • ⚠️ Prompt Augmentation for Self-supervised Text-guided Image Manipulation [Paper]
    • ICLR
      • Guiding Instruction-based Image Editing via Multimodal Large Language Models [Paper] [Code] [Project]
      • The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing [Paper] [Code] [Project]
      • Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators [Paper] [Code] [Project]
      • Object-Aware Inversion and Reassembly for Image Editing [Paper] [Code] [Project]
      • Noise Map Guidance: Inversion with Spatial Context for Real Image Editing [Paper]
    • AAAI
      • Tuning-Free Inversion-Enhanced Control for Consistent Image Editing [Paper]
      • BARET: Balanced Attention based Real image Editing driven by Target-text Inversion [Paper]
      • Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference [Paper]
      • High-Fidelity Diffusion-based Image Editing [Paper]
      • AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing [Paper]
      • ⚠️ TexFit: Text-Driven Fashion Image Editing with Diffusion Models [Paper]
    • arXiv
      • An Item is Worth a Prompt: Versatile Image Editing with Disentangled Control [Paper] [Code]
      • StableDrag: Stable Dragging for Point-based Image Editing [Paper]
      • One-Dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications [Paper] [Code] [Project]
  • Year 2023
  • Year 2022
    • CVPR
      • DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation [Paper] [Code]

<🎯Back to Top>

Text Image Generation

  • Year 2024
    • arXiv
    • CVPR
      • ⚠️ SceneTextGen: Layout-Agnostic Scene Text Image Synthesis with Integrated Character-Level Diffusion and Contextual Consistency [Paper]

<🎯Back to Top>

Datasets

  • Microsoft COCO: Common Objects in Context [Paper] [Dataset]
  • Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning [Paper] [Dataset]
  • LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models [Paper] [Dataset]

<🎯Back to Top>

Toolkits

Name Website Description
Stable Diffusion WebUI link Built based on Gradio, deployed locally to run Stable Diffusion checkpoints, LoRA weights, ControlNet weights, etc.
ComfyUI link Deployed locally to enable customized workflows with Stable Diffusion
Civitai link Websites for community Stable Diffusion and LoRA checkpoints

<🎯Back to Top>

Q&A

  • Q: The conference sequence of this paper list?
    • This paper list is organized according to the following sequence:
      • CVPR
      • ICCV
      • ECCV
      • WACV
      • NeurIPS
      • ICLR
      • ICML
      • ACM MM
      • SIGGRAPH
      • AAAI
      • arXiv
      • Others
  • Q: What does Others refers to?
    • Some of the following studies (e.g., Stable Casacade) does not publish their technical report on arXiv. Instead, they tend to write a blog in their official websites. The Others category refers to such kind of studies.

<🎯Back to Top>

References

The reference.bib file summarizes bibtex references of up-to-date image inpainting papers, widely used datasets, and toolkits. Based on the original references, I have made the following modifications to make their results look nice in the LaTeX manuscripts:

  • Refereces are normally constructed in the form of author-etal-year-nickname. Particularly, references of datasets and toolkits are directly constructed as nickname, e.g., imagenet.
  • In each reference, all names of conferences/journals are converted into abbreviations, e.g., Computer Vision and Pattern Recognition -> CVPR.
  • The url, doi, publisher, organization, editor, series in all references are removed.
  • The pages of all references are added if they are missing.
  • All paper names are in title case. Besides, I have added an additional {} to make sure that the title case would also work well in some particular templates.

If you have other demands of reference formats, you may refer to the original references of papers by searching their names in DBLP or Google Scholar.

<🎯Back to Top>

Star History

Star History Chart

<🎯Back to Top>