/PIA

PIA, your Personalized Image Animator. Animate your images by text prompt, combing with Dreambooth, achieving stunning videos. PIA，你的个性化图像动画生成器，利用文本提示将图像变为奇妙的动画

Primary LanguagePythonApache License 2.0Apache-2.0

PIA：Personalized Image Animator

PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models

Yiming Zhang†, Zhening Xing†, Yanhong Zeng, Youqing Fang, Kai Chen*

(*Corresponding Author, †Equal Contribution)

arXiv Project Page Open in OpenXLab Third Party Colab HuggingFace Model Open in HugginFace

You may also want to try other project from our team: MMagic

PIA is a personalized image animation method which can generate videos with high motion controllability and strong text and image alignment.

What's New

[2024/01/03] Add third-party Colab!

[2023/12/28] PIA can animate a 1024x1024 image with just 16GB of GPU memory with scaled_dot_product_attention!

[2023/12/25] HuggingFace demo is available now! 🤗 Hub

[2023/12/22] Release the model and demo of PIA. Try it to make your personalized movie!

Setup

Prepare Environment

Use the following command to install Pytorch==2.0.0 and other dependencies:

conda env create -f environment-pt2.yaml
conda activate pia

If you want to use lower version of Pytorch (e.g. 1.13.1), you can use the following command:

conda env create -f environment.yaml
conda activate pia

We strongly recommand you to use Pytorch==2.0.0 which supports scaled_dot_product_attention for memory-efficient image animation.

Download checkpoints

  • Download the Stable Diffusion v1-5
    • git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 models/StableDiffusion/
  • Download Personalized Models
    • bash download_bashscripts/1-RealisticVision.sh
bash download_bashscripts/2-RcnzCartoon.sh
bash download_bashscripts/3-MajicMix.sh
  • Download PIA
    • bash download_bashscripts/0-PIA.sh

    You can also download pia.ckpt through link on Google Drive or HuggingFace.

    Put checkpoints as follows:

    └── models
    ├── DreamBooth_LoRA
    │   ├── ...
    ├── PIA
    │   ├── pia.ckpt
    └── StableDiffusion
        ├── vae
        ├── unet
        └── ...

    Usage

    Image Animation

    Image to Video result can be obtained by:

    python inference.py --config=example/config/lighthouse.yaml
python inference.py --config=example/config/harry.yaml
python inference.py --config=example/config/majic_girl.yaml

    Run the command above, you will get:

    Input Image

    lightning, lighthouse

    sun rising, lighthouse

    fireworks, lighthouse

    Input Image

    1boy smiling

    1boy playing the magic fire

    1boy is waving hands

    Input Image

    1girl is smiling

    1girl is crying

    1girl, snowing

    Motion Magnitude

    You can control the motion magnitude through the parameter magnitude:

    python inference.py --config=example/config/xxx.yaml --magnitude=0 # Small Motion
python inference.py --config=example/config/xxx.yaml --magnitude=1 # Moderate Motion
python inference.py --config=example/config/xxx.yaml --magnitude=2 # Large Motion

    Examples:

    python inference.py --config=example/config/labrador.yaml
python inference.py --config=example/config/bear.yaml
python inference.py --config=example/config/genshin.yaml

    Input Image
    & Prompt

    Small Motion

    Moderate Motion

    Large Motion
    a golden labrador is running
    1bear is walking, ...
    cherry blossom, ...

    Style Transfer

    To achieve style transfer, you can run the command(Please don't forget set the base model in xxx.yaml):

    Examples:

    python inference.py --config example/config/concert.yaml --style_transfer
python inference.py --config example/config/ania.yaml --style_transfer

    Input Image
    & Base Model

    1man is smiling

    1man is crying

    1man is singing
    Realistic Vision
    RCNZ Cartoon 3d

    1girl smiling

    1girl open mouth

    1girl is crying, pout
    RCNZ Cartoon 3d

    Loop Video

    You can generate loop by using the parameter --loop

    python inference.py --config=example/config/xxx.yaml --loop

    Examples:

    python inference.py --config=example/config/lighthouse.yaml --loop
python inference.py --config=example/config/labrador.yaml --loop

    Input Image

    lightning, lighthouse

    sun rising, lighthouse

    fireworks, lighthouse

    Input Image

    labrador jumping

    labrador walking

    labrador running

    AnimateBench

    We have open-sourced AnimateBench on HuggingFace which includes images, prompts and configs to evaluate PIA and other image animation methods.

    Contact Us

    Yiming Zhang: zhangyiming@pjlab.org.cn

    Zhening Xing: xingzhening@pjlab.org.cn

    Yanhong Zeng: zengyanhong@pjlab.org.cn

    Acknowledgements

    The code is built upon AnimateDiff, Tune-a-Video and PySceneDetect