Transform your text into captivating, lipsynced animated videos effortlessly, with multilingual support including a wide range of Indian languages.
Loom.Message.-.11.June.2023.mp4
video.mp4
20230611101740_output.mp4
output.mp4
- Project Idea
- Our Approach
- Challenges We Faced
- Technologies We Used
- Network Diagram
- Installation and Setup Guide
Animated videos are loved by almost everyone but the time and expertise required to make them is quite high. Chalchitra.ai was born out of the need to bridge the gap between text-based content and engaging videos. We recognized the demand for synchronized lip-synced animated videos with multilingual support, particularly in Indian languages, empowering content creators, educators, and businesses to convey their messages effectively and captivate their audiences.
-
Transform your text into captivating short animated videos effortlessly. Chalchitra.ai brings your words to life with vibrant visuals and synchronized lip movements.
-
Chalchitra.ai goes beyond language barriers. It offers extensive multilingual support, including a diverse range of Indian languages.
Be at an ad film, a storyboard, a short reel or a quick video for your product, Chalchitra.ai has got you covered
-
Digital Advertising: Generate attention-grabbing animated ads with vibrant visuals effectively conveying your marketing message to the target audience, in a fraction of a time.
-
E-Learning and Education: Enhance the learning experience with visually appealing animated videos that explain complex topics, aided by multilingual support.
-
Language Learning: Aid language learners by creating animated videos demonstrating correct pronunciation, lip movements, and contextual usage of words and phrases in different languages.
-
Entertainment and Media: Produce animated videos for entertainment purposes, such as funny skits, comedic dialogues, or animated web series.
-
News and Information Sharing: Present news updates, information, or public announcements in visually appealing animated videos, grabbing attention and making information easily digestible.
-
Personalized Greetings and Invitations: Add a personal touch to greetings, invitations, or special messages by transforming them into animated videos, delivering heartfelt messages.
Text Manipulation with GPT-3.5 Turbo: When a user submits their text and preferred language, the text is transformed into a script and returned in the JSON format with several attributes.
Image Generation using Stable Diffusion: With the structured JSON, the relevant details are inputted into the stable diffusion model. The prompts are manipulated to ensure that the generated images align seamlessly with the intended narrative and artistic style.
Speech Translation using Google Cloud Translation API: The character's dialogues are translated into the language selected by the user.
Audio generation using Google Cloud Text to Speech API: The audio dialogues of the character are created in the specified language using the Wavenet model, taking into consideration the character's gender and region.
Lip syncing using SadTalker: Stylized audio-driven single image talking face animation is achieved using SadTalker by supplying an image and the corresponding dialogue audio.
Stitching all the generated videos: All the generated videos are combined into a single mp4 file using FFMPEG.
- Finding appropriate pre-trained models for our use cases
- Getting the SadTalker model up and running on our local system
- Stitching all the videos together
- OpenAI's ChatGPT
- Stable Diffusion
- Wavenet (Google Cloud Text to Speech API)
- Google Cloud Translation API
- SadTalker
- Flask
- Next.js
- FastAPI
- FFMPEG
To get started with Comicify.ai, follow these steps:
- Clone the repo:
git clone https://github.com/pranav2305/chalchitra.ai.git
- Go to project directory
cd chalchitra.ai
- Change directory to client
cd client
- Install npm packages
npm i
- Run the dev server
npm run dev
- Change directory to server
cd server
- Create a .env file from .env.example
cp .env.example .env
. - Update the API keys in .env:
a.OPEN_AI_API = '<your-api-key>'
. Follow these instructions to obtain your key from OpenAI.
b.STABLE_DIFFUSION_API = '<your-api-key>'
. Follow these instructions to obtain your key from Dream Studio. - Create a virtual environment and activate it.
python -m venv env
- Install the requirements
python -r requirement.txt
- Get your Google Applications Credentials json file and activate it
export GOOGLE_APPLICATION_CREDENTIALS=<path_to_json_file>
- Run the flask server
flask --app main run
Refer the SadTalker README.md for setup instructions