This repo holds a modularized version of Disco Diffusion for use with ComfyUI.
- Clone this repository into your
ComfyUI/custom_nodes
folder withgit clone --recursive https://github.com/space-nuko/ComfyUI-Disco-Diffusion
. - Activate the virtualenv you use with ComfyUI.
- Navigate to the cloned folder and run
pip install -r requirements.txt
. - Start ComfyUI.
- Guided Diffusion Loader - Loads a Guided Diffusion model checkpoint. All models are automatically downloaded to your
ComfyUI/models/Disco-Diffusion
folder. - OpenAI CLIP Loader - Loads the OpenAI/OpenCLIP format CLIP models. All models are automatically downloaded and cached. You can also use ComfyUI's built-in CLIP loader nodes. The recommended model for use with the built-in nodes is
clip-vit-large-patch14
, found here. - Disco Diffusion - The main sampler node.
- Disco Diffusion Extra Settings - Optional settings you can pass into the Disco Diffusion node.
The simplest usage is to connect the Guided Diffusion Loader and OpenAI CLIP Loader nodes into a Disco Diffusion node, then hook the Disco Diffusion node up to a Save Image node.
Alternatively, you can substitute the OpenAI CLIP Loader for ComfyUI's CLIP Loader and CLIP Vision Loader, however in this case you need to copy the CLIP model you use into both the clip
and clip_vision
subfolders under your ComfyUI/models
folder, because ComfyUI can't load both at once from the same model file.
The Disco Diffusion node uses a special syntax for writing prompts, which is shown below.
; You can write comments like this.
0:A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea of blood by greg rutkowski and thomas kinkade. Trending on artstation.
0:yellow color scheme
100:This set of prompts start at frame 100.
100:This prompt has weight five:5
The leading number in front of each prompt indicates which step number in the generation the prompt should begin being used on. More than one prompt with the same number is concatenated into a list, ordered from top to bottom. This lets you write prompts that are longer than 77 tokens total, as that is the maximum context length that CLIP is capable of. (By contrast, stable-diffusion-webui automatically splits up longer prompts for you)
The trailing number at the end of each prompt is optional and controls the weight of the prompt. You can also have negative weights, but the sum of all weights cannot equal 0.
Read the Disco Diffusion Cheatsheet for more information about the parameters available. Also check out the EZ Charts Gudie