/Awesome-SD-Inference

📖A small curated list of Awesome SD/DiT/ViT/Diffusion Inference with Distributed/Caching/Sampling: DistriFusion, PipeFusion, AsyncDiff, DeepCache, Block Caching etc.

GNU General Public License v3.0GPL-3.0

📒Awesome-SD-Inference

📒A small curated list of Awesome SD/DiT/Diffusion Inference with Distributed/Caching/Sampling. For Awesome LLM Inference, please check 📖Awesome-LLM-Inference

🤖Contents

©️Citations

@misc{Awesome-SD-Inference@2024,
  title={Awesome-SD-Inference: A small curated list of Awesome SD/DiT/ViT/Diffusion Inference with Distributed/Caching/Sampling.},
  url={https://github.com/DefTruth/Awesome-SD-Inference},
  note={Open-source software available at https://github.com/DefTruth/Awesome-SD-Inference},
  author={DefTruth},
  year={2024}
}

📙Awesome SD Inference with Sampling

Date Title Paper Code Recom
2020.06 🔥[DDPM] Denoising Diffusion Probabilistic Models(@UC Berkeley) [pdf] [diffusion] ⭐️⭐️
2020.10 🔥[DDIM] DENOISING DIFFUSION IMPLICIT MODELS(@cs.stanford.edu) [pdf] ⚠️ ⭐️⭐️
2022.02 🔥[PNDM] PSEUDO NUMERICAL METHODS FOR DIFFUSION MODELS ON MANIFOLDS(@) [pdf] [PNDM] ⭐️⭐️
2022.02 🔥[DPM-Solver] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps(@Cheng Lu) [pdf] [dpm-solver] ⭐️⭐️
2022.11 🔥[DPM-Solver++] DPM-SOLVER++: FAST SOLVER FOR GUIDED SAMPLING OF DIFFUSION PROBABILISTIC MODELS(@Cheng Lu) [pdf] [dpm-solver] ⭐️⭐️
2023.10 🔥[DPM-Solver-v3] DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics(@Kaiwen Zheng) [pdf] [DPM-Solver-v3] ⭐️⭐️
2023.11 🔥[Parallel Sampling] Parallel Sampling of Diffusion Models(@Stanford University) [pdf] [paradigms] ⭐️⭐️
2023.11 🔥[SAMPLER SCHEDULER] SAMPLER SCHEDULER FOR DIFFUSION MODELS(@sysu) [pdf] ⚠️ ⭐️⭐️
2024.02 🔥[Parallel Sampling] Accelerating Parallel Sampling of Diffusion Models(@Zhiwei Tang) [pdf] [ParaTAA-Diffusion] ⭐️⭐️
2024.01 🔥[YONOS] You Only Need One Step: Fast Super-Resolution with Stable Diffusion via Scale Distillation(@Samsung AI) [pdf] ⚠️ ⭐️⭐️
2024.01 🔥[S^2-DM] S^2-DMs: Skip-Step Diffusion Models(@Yixuan Wang) [pdf] ⚠️ ⭐️⭐️
2024.08 🔥[StepSaver] StepSaver: Predicting Minimum Denoising Steps for Diffusion Model Image Generation(@intel) [pdf] ⚠️ ⭐️⭐️
2024.09 🔥[DC-Solver] DC-Solver: Improving Predictor-Corrector Diffusion Sampler via Dynamic Compensation(@Tsinghua University) [pdf] [DC-Solver] ⭐️⭐️

📙Awesome SD Inference with Caching

  • UNet Based (DeepCache)
image
  • DiT Based (Fast-Forward Caching)
image
Date Title Paper Code Recom
2023.05 🔥🔥[Cache-Enabled Sparse Diffusion] Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference(@pku.edu.cn etc) [pdf] ⚠️ ⭐️⭐️
2023.12 🔥🔥[DeepCache] DeepCache: Accelerating Diffusion Models for Free(@nus.edu) [pdf] [DeepCache] ⭐️⭐️
2023.12 🔥🔥[Block Caching] Cache Me if You Can: Accelerating Diffusion Models through Block Caching(@Meta GenAI etc) [pdf] ⚠️ ⭐️⭐️
2023.12 🔥🔥[Approximate Caching] Approximate Caching for Efficiently Serving Diffusion Models(@Adobe) [pdf] ⚠️ ⭐️⭐️
2024.06 🔥🔥[Layer Caching] Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching(@nus.edu) [pdf] [learning-to-cache] ⭐️⭐️
2024.07 🔥[ElasticCache-LVLM] Efficient Inference of Vision Instruction-Following Models with Elastic Cache(@Tsinghua University etc) [pdf] [ElasticCache] ⭐️
2024.07 🔥🔥[Fast-Forward Caching(DiT)] FORA: Fast-Forward Caching in Diffusion Transformer Acceleration(@microsoft.com etc) [pdf] [FORA] ⭐️⭐️
2024.07 🔥🔥[Faster I2V Generation] Faster Image2Video Generation: A Closer Look at CLIP Image Embedding’s Impact on Spatio-Temporal Cross-Attentions(@Ashkan Taghipour etc) [pdf] ⚠️ ⭐️⭐️
2024.04 🔥🔥[T-GATE V1] Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion Models(@Wentian Zhang etc) [pdf] [T-GATE] ⭐️⭐️
2024.04 🔥🔥[T-GATE V2] Faster Diffusion via Temporal Attention Decomposition(@Haozhe Liu etc) [pdf] [T-GATE] ⭐️⭐️

📙Awesome SD Distributed Inference with Multi-GPUs

  • UNet Based: Displaced Patch parallelism (DistriFusion)
image
  • DiT Based: Displaced Patch parallelism (PipeFusion)
image
Date Title Paper Code Recom
2024.02 🔥🔥[DistriFusion] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models(@MIT etc) [pdf] [distrifuser] ⭐️⭐️
2024.05 🔥🔥[PipeFusion] PipeFusion: Displaced Patch Pipeline Parallelism for Inference of Diffusion Transformer Models(@Tencent etc) [pdf] [xDiT] ⭐️⭐️
2024.06 🔥🔥[AsyncDiff] AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising(@nus.edu) [pdf] [AsyncDiff] ⭐️⭐️
2024.05 🔥🔥[TensorRT-LLM SDXL] SDXL Distributed Inference with TensorRT-LLM and synchronous comm(@Zars19) [pdf] [SDXL-TensorRT-LLM] ⭐️⭐️
2024.06 🔥🔥[Clip Parallelism] Video-Infinity: Distributed Long Video Generation(@nus.edu) [pdf] [Video-Infinity] ⭐️⭐️
2024.05 🔥🔥[FIFO-Diffusion] FIFO-Diffusion: Generating Infinite Videos from Text without Training(@Seoul National University) [pdf] [FIFO-Diffusion] ⭐️⭐️

📙Other Awesome SD Inference Paper with codes

Date Title Paper Code Recom
2024.06 🔥[DiTFastAttn] DiTFastAttn: Attention Compression for Diffusion Transformer Models(@Zhihang Yuan etc) [pdf] [DiTFastAttn] ⭐️⭐️
2024.08 🔥[Transfusion] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model(@meta) [pdf] [transfusion-pytorch] ⭐️⭐️
2024.08 🔥[VQ4DiT] VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformers(@ZJU) [pdf] ⚠️ ⭐️⭐️
2024.08 🔥[LBQ] Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models(@toronto.edu) [pdf] ⚠️ ⭐️⭐️
2024.08 🔥[EE-Diffusion] A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models(@KAIST AI) [pdf] [ee-diffusion] ⭐️⭐️
2024.08 🔥[TFM-PTQ] Temporal Feature Matters: A Framework for Diffusion Model Quantization(@SenseTime) [pdf] ⚠️ ⭐️⭐️
2024.08 🔥[Diffusion-RWKV] Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models(@Zhengcong Fei) [pdf] [Diffusion-RWKV] ⭐️⭐️
2024.09 🔥[LinFusion] LINFUSION: 1 GPU, 1 MINUTE, 16K IMAGE(@NUS) [pdf] [LinFusion] ⭐️⭐️

©️License

GNU General Public License v3.0

🎉Contribute

Welcome to star & submit a PR to this repo!

Star History Chart