🐣 Please follow me for new updates https://twitter.com/camenduru
🔥 Please join our discord server https://discord.gg/k5BwmmvJJU

The aim of this project is to develop an open-source text-to-video model and create an easy-to-use interface for artists.

We will decide on the project name later (code name: text-to-video-model 😋)

Open In Colab

Performance

  • System RAM: ~8.5 GB

  • VRAM: ~11 GB

  • Model Size: ~4.1G

Screenshot 2023-08-23 130300

Potat 1️⃣ (Prototype Model)

243292723-fa703668-a931-41e1-8bcf-19c72203980b

  • Open-Source 1024x576 Text To Video Model 🥳

  • Trained with 1xA100 (40GB)

  • 2197 clips, 68388 tagged frames ( salesforce/blip2-opt-6.7b-coco )

  • train_steps: 10000

Examles

Prompt: Octopus under the ocean.

download_31.mp4

📦 Dataset Preparation

https://github.com/microsoft/XPretrain/tree/main/hd-vila-100m (HD-VILA-100M Dataset)
http://toflow.csail.mit.edu/ (Vimeo-90k Dataset)
https://github.com/m-bain/webvid
https://github.com/ExponentialML/Video-BLIP2-Preprocessor
https://github.com/Breakthrough/PySceneDetect

🍱 Finetuning

https://github.com/guoyww/animatediff
https://github.com/showlab/Tune-A-Video
https://github.com/ExponentialML/Text-To-Video-Finetuning
https://www.modelscope.cn/models/damo/text-to-video-synthesis

📅 Management Framework & UI & Database

https://github.com/spring-projects/spring-framework
https://github.com/gradio-app/gradio
https://github.com/mongodb/mongo

🎴 Demo UI & Test

https://platform.stability.ai/docs/api-reference
https://github.com/angular/angular
https://colab.research.google.com/
https://github.com/jupyter/notebook

🎎 Who we are?

https://twitter.com/camenduru (at this point just me 😥)
Camenduru is a computer engineer who has worked with game studios (Unity, Unreal Engine, VR (Virtual Reality), PC and Mobile games).
Camenduru is also the creator of several ML models and notebooks, which can be found at https://github.com/camenduru and https://huggingface.co/camenduru

💰 Sponsor

https://100.builders/
https://twitter.com/withBackdrop
https://base.org/
https://stability.ai/
https://www.seedclub.xyz/
https://www.sismo.io/
https://www.artizen.fund/
https://layerx.xyz/

image