AnimateDiff integration for ComfyUI, adapts from sd-webui-animatediff. Please read the original repo README for more information.
- Clone this repo into
custom_nodes
folder. - Download motion modules and put them under
comfyui-animatediff/models/
.
- Original modules: Google Drive | HuggingFace | CivitAI | Baidu NetDisk
- Community modules: manshoety/AD_Stabilized_Motion | CiaraRowles/TemporalDiff
- AnimateDiff v2 mm_sd_v15_v2.ckpt
Download motion LoRAs and put them under comfyui-animatediff/loras/
folder.
Note: LoRAs only work with AnimateDiff v2 mm_sd_v15_v2.ckpt module.
![image](https://private-user-images.githubusercontent.com/133728487/270393962-7a9f62f7-702e-48a4-934c-bbfe1e23aff2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjcwMzkzOTYyLTdhOWY2MmY3LTcwMmUtNDhhNC05MzRjLWJiZmUxZTIzYWZmMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNTdhZjJhYTgzNDQwMmUwOThlN2Y2OTc2OGEwNjM5NTNjNTRjNmQzMTZjYjA1OTRiNGMzOGE0MmYyMTRlYTkxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.NlYjGmBc6ziaJ4qfBISrEZwtzXrTWK_tdXR_IjhC72g)
Workflow: lora.json
Samples:
![]() |
![]() |
![]() |
![]() |
The sliding window feature enables you to generate GIFs without a frame length limit. It divides frames into smaller batches with a slight overlap. This feature is activated automatically when generating more than 16 frames. To modify the trigger number and other settings, utilize the SlidingWindowOptions
node. See the sample workflow bellow.
![image](https://private-user-images.githubusercontent.com/133728487/268482819-9d756d01-ea45-4d1c-8e48-56f2725c7ca1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODE5LTlkNzU2ZDAxLWVhNDUtNGQxYy04ZTQ4LTU2ZjI3MjVjN2NhMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1mZWQxMDk0MWNlYmE1NjdmMzNjMDk1MzI3YTU5ZGUwYjM2ODIzYThjOTJkZThhOGQ0MTk5ODEwYmJjZDJkNGIyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.PdUDo3eOhVkMbGHa47Us5nwvxhP5eigmqxp9HbEJQWA)
- Mostly the same with
KSampler
motion_module
: useAnimateDiffLoader
to load the motion moduleinject_method
: should left defaultframe_number
: animation lengthlatent_image
: You can pass anEmptyLatentImage
sliding_window_opts
: custom sliding window options
![image](https://private-user-images.githubusercontent.com/133728487/269430269-a352195d-f40c-494d-bd3d-30ee88174b88.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMwMjY5LWEzNTIxOTVkLWY0MGMtNDk0ZC1iZDNkLTMwZWU4ODE3NGI4OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00ZTFmZTJlYjRkZjlmNjdmMTJkZDcwYjI3YTk3YTU2NDk1NWY3MTUxMGQ3M2UzYThmNTFmZjYxNmQzYTFiM2FmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.DSPwOpSCAPozpciwYkh4QFrmNiRbFVLr_hHNqsVEcKw)
- Combine GIF frames and produce the GIF image
frame_rate
: number of frame per secondloop_count
: use 0 for infinite loopsave_image
: should GIF be saved to diskformat
: supportsimage/gif
,image/webp
(better compression),video/webm
,video/h264-mp4
,video/h265-mp4
. To use video formats, you'll need ffmpeg installed and available inPATH
![image](https://private-user-images.githubusercontent.com/133728487/268482889-381c5acc-06ef-43da-ada0-3dc76f37a3e4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDgyODg5LTM4MWM1YWNjLTA2ZWYtNDNkYS1hZGEwLTNkYzc2ZjM3YTNlNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1mN2EyZGEwODYxZDc2Y2EzMmY0NmViMDVkMDhiODk4Zjg5MzAwNzRmYTRlNTFhNWQ0ZmI3ZmI2Yzk1MWQ4MjdmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.E4KccTaAzmQvm3mC5trQU0WB-GvWAa8AaOzzdl3Yjm4)
Custom sliding window options
context_length
: number of frame per window. Use 16 to get the best results. Reduce it if you have low VRAM.context_stride
:- 1: sampling every frame
- 2: sampling every frame then every second frame
- 3: sampling every frame then every second frame then every third frames
- ...
context_overlap
: overlap frames between each window sliceclosed_loop
: make the GIF a closed loop, will add more sampling step
![image](https://private-user-images.githubusercontent.com/133728487/269431763-6679a8dd-bf96-419f-8934-ea2b046dd23c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMxNzYzLTY2NzlhOGRkLWJmOTYtNDE5Zi04OTM0LWVhMmIwNDZkZDIzYy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01ZDM1ZTBkMjcwMTYwMDQ0NDQ4ZjViMjJlOTc3MzlkYTcyZjNkZTEzODU2NDNmNzI1ZjhlYmMzZTgyZjBkZjFkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.kIzNRy_OgQBH7_21Bk1-hlhYstZvKHAWdyQ27fJ-LmU)
Load GIF or video as images. Usefull to load a GIF as ControlNet input.
frame_start
: Skip some begining frames and start atframe_start
frame_limit
: Only takeframe_limit
frames
![image](https://private-user-images.githubusercontent.com/133728487/269429598-684176d5-6369-4a27-9f33-e721e0fe1876.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDI5NTk4LTY4NDE3NmQ1LTYzNjktNGEyNy05ZjMzLWU3MjFlMGZlMTg3Ni5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02OWViMmVjNGUwNWUyYzBhY2ZhMzk5MDM0MWFlNGMxNGNmNGQ3N2JmMjI5YTgzMjQ4YTNmYjI4N2E1M2NjM2Q1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.VFOD2rH9BysBIR2tDpEKM_7FkGVeNktO6XiZnJN3q10)
![image](https://private-user-images.githubusercontent.com/133728487/268311503-b7164539-bc58-4ef9-b178-d914e833805e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4MzExNTAzLWI3MTY0NTM5LWJjNTgtNGVmOS1iMTc4LWQ5MTRlODMzODA1ZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00NjFhMDE0ZjY4ZDEzOWUzN2M1NGQ0NzgxYjJiODMzZGEzMWIyNzFiODNhNjQ2ODU4MTE0MDg4YjRmNTZjYmRhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.RQhbe-l2tEvWETjMtqqooKAu23lgKM6pfVpXtbOWRVw)
Workflow: simple.json
Samples:
![image](https://private-user-images.githubusercontent.com/133728487/269432912-0f8bfb87-83cb-4119-9777-e3948ec0cb5c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY5NDMyOTEyLTBmOGJmYjg3LTgzY2ItNDExOS05Nzc3LWUzOTQ4ZWMwY2I1Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04YWNiMDA4OGFhYjZjYzQ1NmQ4Y2FhZmZmZDY5OWU5NjdjZTgyMDQzNTEzMGFhNzQ5NzYzZDdkNDNjNTY4ZjAwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.3In05PauvP11k8N1jI1VF2n9roc4J72NxMnX_F14jrE)
Workflow: sliding-window.json
Samples:
![]() |
![]() |
Upscale latent output using LatentUpscale
then do a 2nd pass with AnimateDiffSampler
.
![image](https://private-user-images.githubusercontent.com/133728487/268485612-987a1c5a-c1f8-4b24-8c62-f14496261d6c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA1MDM1MjMsIm5iZiI6MTcyMDUwMzIyMywicGF0aCI6Ii8xMzM3Mjg0ODcvMjY4NDg1NjEyLTk4N2ExYzVhLWMxZjgtNGIyNC04YzYyLWYxNDQ5NjI2MWQ2Yy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzA5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcwOVQwNTMzNDNaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT02NjMwNGM2MjE5M2E5NWI3NTdkYjFmMWU4MDRkM2QwNjU3NzhiNzg5YTBlNDQ4OTc5NGU3MzRjYjM2NzJiZjg5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.W-d1du0atLR2fGzubOGiZ3xfQBgO-ivy8wPuI9jDJ5E)
Workflow: latent-upscale.json
You will need following additional nodes:
- Kosinkadink/ComfyUI-Advanced-ControlNet: Apply different weight for each latent in batch
- Fannovel16/comfyui_controlnet_aux: ControlNet preprocessors
- Use
LatentKeyframe
andTimestampKeyframe
from ComfyUI-Advanced-ControlNet to apply diffrent weights for each latent index. - Use 2 controlnet modules for two images with weights reverted.
Workflow: cn-2images.json
Samples:
![]() |
![]() |
![]() |
![]() |
Using a GIF (or video, or a list of images) as ControlNet input.
Workflow: cn-vid2vid.json
Samples:
![]() |
![]() |
It's an xformers
bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. The current workaround is to disable xformers with --disable-xformers
when booting ComfyUI.
Work around:
- Shorter your prompt and negative prompt
- Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output.
- Disable xformers with
--disable-xformers
See: continue-revolution/sd-webui-animatediff#31
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.