jasperan/draw-realtime

Draw stories in Real Time with StreamDiffusion, TTS and ControlNet

StreamDiffusion

English | 日本語

StreamDiffusion: A Pipeline-Level Solution for Real-Time Interactive Generation

StreamDiffusion is an innovative diffusion pipeline designed for real-time interactive generation. It introduces significant performance enhancements to current diffusion-based image generation techniques.

Key Features

1. Stream Batch

   • Streamlined data processing through efficient batch operations.

2. Residual Classifier-Free Guidance - Learn More

   • Improved guidance mechanism that minimizes computational redundancy.

3. Stochastic Similarity Filter - Learn More

   • Improves GPU utilization efficiency through advanced filtering techniques.

4. IO Queues

   • Efficiently manages input and output operations for smoother execution.

5. Pre-Computation for KV-Caches

   • Optimizes caching strategies for accelerated processing.

6. Model Acceleration Tools

   • Utilizes various tools for model optimization and performance boost.

When images are produced using our proposed StreamDiffusion pipeline in an environment with GPU: RTX 4090, CPU: Core i9-13900K, and OS: Ubuntu 22.04.3 LTS.

model	Denoising Step	fps on Txt2Img	fps on Img2Img
SD-turbo	1	106.16	93.897
LCM-LoRA + KohakuV2	4	38.023	37.133

Installation

Step0: clone this repository

git clone https://github.com/cumulo-autumn/StreamDiffusion.git

Step1: Make Environment

You can install StreamDiffusion via pip, conda, or Docker(explanation below).

conda create -n streamdiffusion python=3.10
conda activate streamdiffusion

OR

python -m venv .venv
# Windows
.\.venv\Scripts\activate
# Linux
source .venv/bin/activate

Step2: Install PyTorch

Select the appropriate version for your system.

CUDA 11.8

pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu118

CUDA 12.1

pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu121

details: https://pytorch.org/

Step3: Install StreamDiffusion

For User

Install StreamDiffusion

#for Latest Version (recommended)
pip install git+https://github.com/cumulo-autumn/StreamDiffusion.git@main#egg=streamdiffusion[tensorrt]


#or


#for Stable Version
pip install streamdiffusion[tensorrt]

Install TensorRT extension

python -m streamdiffusion.tools.install-tensorrt

(Only for Windows) You may need to install pywin32 additionally, if you installed Stable Version(pip install streamdiffusion[tensorrt]).

pip install --force-reinstall pywin32

For Developer

python setup.py develop easy_install streamdiffusion[tensorrt]
python -m streamdiffusion.tools.install-tensorrt

Docker Installation (TensorRT Ready)

git clone https://github.com/cumulo-autumn/StreamDiffusion.git
cd StreamDiffusion
docker build -t stream-diffusion:latest -f Dockerfile .
docker run --gpus all -it -v $(pwd):/home/ubuntu/streamdiffusion stream-diffusion:latest

Quick Start

You can try StreamDiffusion in examples directory.

Acknowledgements

jasperan