/animatediff_exp

Experiments with Animate diff and LoRA

Primary LanguageJupyter NotebookMIT LicenseMIT

🎥 Animate Diff Experiment for Advertisement GIFs

Welcome to the Animate Diff for Advertisement GIFs repository! This project uses a combination of OpenAI's GPT and the Animate Diff model to create promotional GIFs from text descriptions. 📝✨

🚀 Features

  • Text-to-Video Generation: Generate captivating GIFs from product descriptions.
  • **OpenAI GPT **: Uses GPT to create compelling scripts and prompts.
  • LoRA Weights: Incorporate specialized LoRA weights for GIF generation of specific brands and personalities.

📈 Experiments and Results

Here are the results of the experiments conducted using various LoRA weights:

🥤 Coca-Cola Ad GIFs

Generated GIFs with stable diffusion v1.5 (base-model), AnimateLCM (motion adapter) and tilt-up LoRA Generated GIFs with epicRealism (base-model), AnimateDiff-Lightning (motion adapter) and zoom-in LoRA
Coca-Cola GIF Coca-Cola alt GIF
Coca-Cola 2 GIF Coca-Cola alt 2 GIF

💪 Coca-Cola & Dwayne Johnson Ad GIFs

Generated GIFs with stable diffusion v1.5 (base-model), AnimateLCM (motion adapter) and tilt-up LoRA Generated GIFs with epicRealism (base-model), AnimateDiff-Lightning (motion adapter) and zoom-in LoRA
Coca-Cola & Dwayne Johnson GIF Coca-Cola & Dwayne Johnson alt GIF

🧠 How It Works

  1. Script and Prompt Generation 🎭

    • The first step involves using OpenAI's GPT endpoint to generate a commercial script and a detailed text prompt.
    • Example prompt: "The Rock takes a Coca-Cola bottle and poses with a satisfactory smile, 8k, 4k, UHD, highly detailed"

  2. Editing prompt ✂️

    • The next step involves editing the prompt with LoRA weights specific keywords like "th3rock", "kekokelev2" etc (the keyword used while dream-booth fine-tuning)

  3. GIF Creation 🎞️

    • The edited prompt is then fed into the Animate Diff model, which uses LoRA weights to adapt and produce the corresponding GIF.

  4. Evaluation 🔍

    • Each generated GIF should be manually evaluated for its quality, with adjustments made to the script (scale of each LoRA weight) and prompt as needed.

Future Work 📝

Improving results for multiple LoRA - There are many techniques for combining LoRAs like Ties and Dare, and I will try to explore them

Generating Longer Video sequence - There are some techniques like applying sliding-window to generate longer sequences, I will try to look into this.

🤔 FAQs

Q: Can I use this for any product?

Absolutely! Just tweak the product name and description in the text prompt, add its corresponding LoRA weights and the model will generate a customized GIF for you.

Q: How do I add new LoRA weights?

You can train new LoRA weights for specific objects or you can use sites like Civitai where you can find many pre-trained LoRA weights for different objects and styles

📫 Contributions

Contributions are welcome! Please open an issue or submit a pull request if you have any suggestions or improvements.

🙏 Acknowledgements

📜 License

This project is licensed under the MIT License. See the LICENSE file for details.

Thank you for your interest! I hope you enjoy using the animated advertisement GIF generator. 🎉

Star the Repo

Feel free to reach out if you have any questions or need support. Happy GIF-making! 🚀