In this demo we are using websockets to stream incoming audio from a radio broadcast to ws://api.simli.ai/LipsyncStream
which returns lipsynced video and audio frames to be displayed on frontend
Websocket schema: Click here
sequenceDiagram
participant Client
participant Frontend
participant AudioServer
participant LipsyncServer
Client->>Frontend: Open webpage
activate Frontend
Frontend->>Frontend: Render UI
Frontend->>Frontend: User clicks on Start button
Frontend->>Frontend: AudioContext and VideoContext initialized
Frontend->>Frontend: WebSocket connection created for audio
Frontend->>Frontend: WebSocket connection created for lipsync
Frontend->>AudioServer: Connects to audio server
activate AudioServer
AudioServer->>AudioServer: WebSocket connection opened
AudioServer->>Frontend: Connection established
deactivate AudioServer
Frontend->>LipsyncServer: Connects to lipsync server
activate LipsyncServer
LipsyncServer->>LipsyncServer: WebSocket connection opened
LipsyncServer->>Frontend: Connection established
deactivate LipsyncServer
Frontend->>LipsyncServer: Sends metadata
Frontend->>Frontend: Start main loop for audio and video playback
Frontend->>AudioServer: Receive audio broadcast data
AudioServer->>Frontend: Audio data
Frontend->>LipsyncServer: Send Audio data to lipsync
LipsyncServer->>Frontend: Lipsync data
Frontend->>Frontend: Process lipsync data to frames and audio
Frontend->>Frontend: Update audio queue
Frontend->>Frontend: Update video queue
Frontend->>Frontend: Play audio frames queue
Frontend->>Frontend: Play video frames queue
deactivate Frontend
- Install FFMPEG
sudo apt install ffmpeg
- Install python modules
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
- run radio broadcast websocket server
python app.py
- Install FFMPEG
winget install ffmpeg
- Install python modules
pip install -r requirements.txt
- run radio broadcast websocket server
python app.py
Open another terminal
- Install packages
npm install
- Create
.env
and add Simli API Key
NEXT_PUBLIC_SIMLI_API_KEY=YOUR-SIMLI-API-KEY
- Start
npm run dev