AnimateDiff

This repository is the official implementation of AnimateDiff.

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Yuwei Guo, Ceyuan Yang*, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai

*Corresponding Author

Arxiv Report | Project Page

Todo

Code Release
Arxiv Report
GPU Memory Optimization
Gradio Interface

Setup for Inference

Prepare Environment

Our approach takes around 60 GB GPU memory to inference. NVIDIA A100 is recommanded.

git clone https://github.com/guoyww/animatediff.git
cd animatediff

conda create -n animatediff python=3.8
conda activate animatediff

pip install -r requirements.txt

Download Base T2I & Motion Module Checkpoints

We provide two versions of our Motion Module, which are trained on stable-diffusion-v1-4 and finetuned on v1-5 seperately. It's recommanded to try both of them for best results.

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/

bash download_bashscripts/0-MotionModule.sh

You may also directly download the motion module checkpoints from Google Drive, then put them in models/Motion_Module/ folder.

Prepare Personalize T2I

Here we provide inference configs for 6 demo T2I on CivitAI. You may run the following bash scripts to download these checkpoints.

bash download_bashscripts/1-ToonYou.sh
bash download_bashscripts/2-Lyriel.sh
bash download_bashscripts/3-RcnzCartoon.sh
bash download_bashscripts/4-MajicMix.sh
bash download_bashscripts/5-RealisticVision.sh
bash download_bashscripts/6-Tusun.sh
bash download_bashscripts/7-FilmVelvia.sh
bash download_bashscripts/8-GhibliBackground.sh

Inference

After downloading the above peronalized T2I checkpoints, run the following commands to generate animations.

python -m scripts.animate --config configs/prompts/1-ToonYou.yaml
python -m scripts.animate --config configs/prompts/2-Lyriel.yaml
python -m scripts.animate --config configs/prompts/3-RcnzCartoon.yaml
python -m scripts.animate --config configs/prompts/4-MajicMix.yaml
python -m scripts.animate --config configs/prompts/5-RealisticVision.yaml
python -m scripts.animate --config configs/prompts/6-Tusun.yaml
python -m scripts.animate --config configs/prompts/7-FilmVelvia.yaml
python -m scripts.animate --config configs/prompts/8-GhibliBackground.yaml

Gallery

Here we demonstrate several best results we got in previous experiments.

Model：ToonYou

Model：Counterfeit V3.0

Model：Realistic Vision V2.0

Model： majicMIX Realistic

Model：RCNZ Cartoon

Model：FilmVelvia

BibTeX

@misc{guo2023animatediff,
      title={AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning}, 
      author={Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, Bo Dai},
      year={2023},
      eprint={2307.04725},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contact Us

Yuwei Guo: guoyuwei@pjlab.org.cn
Ceyuan Yang: yangceyuan@pjlab.org.cn
Bo Dai: daibo@pjlab.org.cn

Acknowledgements

Codebase built upon Tune-a-Video.

hyojunguy/AnimateDiff