/simli-radio-next-js-demo

Using websockets to stream incoming audio from a radio broadcast to Simli's API /LipsyncStream to process and return lipsynced video and audio frames to be displayed on frontend

Primary LanguageTypeScript

simli-next-js-demo

In this demo we are using websockets to stream incoming audio from a radio broadcast to ws://api.simli.ai/LipsyncStream which returns lipsynced video and audio frames to be displayed on frontend

Websocket schema: Click here

sequenceDiagram
    participant Client
    participant Frontend
    participant AudioServer
    participant LipsyncServer

    Client->>Frontend: Open webpage
    activate Frontend
    Frontend->>Frontend: Render UI
    Frontend->>Frontend: User clicks on Start button
    Frontend->>Frontend: AudioContext and VideoContext initialized
    Frontend->>Frontend: WebSocket connection created for audio
    Frontend->>Frontend: WebSocket connection created for lipsync
    Frontend->>AudioServer: Connects to audio server
    activate AudioServer
    AudioServer->>AudioServer: WebSocket connection opened
    AudioServer->>Frontend: Connection established
    deactivate AudioServer
    Frontend->>LipsyncServer: Connects to lipsync server
    activate LipsyncServer
    LipsyncServer->>LipsyncServer: WebSocket connection opened
    LipsyncServer->>Frontend: Connection established
    deactivate LipsyncServer
    Frontend->>LipsyncServer: Sends metadata
    Frontend->>Frontend: Start main loop for audio and video playback
    Frontend->>AudioServer: Receive audio broadcast data
    AudioServer->>Frontend: Audio data
    Frontend->>LipsyncServer: Send Audio data to lipsync
    LipsyncServer->>Frontend: Lipsync data
    Frontend->>Frontend: Process lipsync data to frames and audio
    Frontend->>Frontend: Update audio queue
    Frontend->>Frontend: Update video queue
    Frontend->>Frontend: Play audio frames queue
    Frontend->>Frontend: Play video frames queue
    deactivate Frontend
Loading

Usage

Terminal 1: (AudioServer)

Linux

  1. Install FFMPEG
sudo apt install ffmpeg
  1. Install python modules
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
  1. run radio broadcast websocket server
python app.py

Windows

  1. Install FFMPEG
winget install ffmpeg
  1. Install python modules
pip install -r requirements.txt
  1. run radio broadcast websocket server
python app.py

Terminal 2: (Frontend)

Open another terminal

  1. Install packages
npm install
  1. Create .env and add Simli API Key
NEXT_PUBLIC_SIMLI_API_KEY=YOUR-SIMLI-API-KEY
  1. Start
npm run dev