/Temporal-Image-AnimateDiff

A retrain of AnimateDiff to be conditional on an init image

Primary LanguagePythonApache License 2.0Apache-2.0

Temporal-Image-AnimateDiff

Introducing an AnimateDiff retrain over the sd-inpainting model in order to take an input image

Examples

Input Prompt Output
stockman Man on a Beach A man on a beach
tay Taylor Swift Taylor Swift
beachw Woman on a Beach 1-a-woman

Setup & Configuration

  1. Download Motion Module

    • Motion module from HuggingFace
    • Place it in the motion_module folder.

  2. Configuration Keys:

    • base: Set this to "models/StableDiffusionInpainting". This should point to the diffuser's inpainting model available here.
    • vae: Use "models/StableDiffusion". This must link to the original 1.5 stable diffusion model due to a diffusers issue. Get the vae from here.
    • path: Specify something like "models/DreamBooth_LoRA/realisticVisionV20_v20.inpainting.safetensors". It must be your Dreambooth model and needs to be an inpainting model. Note: You can convert existing models to inpainting models by adding the difference from the inpainting model and the standard model to any custom model.

  3. Execution:

    python -m scripts.animate --config "configs/prompts/5-RealisticVision.yaml" --image_path "/images/image.png" --W 256 --H 256

Considerations & Recommendations

  • This model currently only works at roughly 256x256 resolutions. Retraining it to 512x512 didn't work for some reason, so you'd be best just upscaling with comfyui for now.

  • In terms of making it work well, consider the prompts are not you telling the model what you want with this, you're guiding the generation for what is in the input image, if the input image and the prompts do not align, it will not work.

  • You may have to try a few seeds per generation to get a nice image, it's a tiny bit unreliable.

Acknowledgements

Special thanks to the developers behind AniamteDiff. https://github.com/guoyww/AnimateDiff.