truonganhhoang/whispering

Whispering Tiger - OpenAI's whisper with OSC and Websocket support. Allowing live transcription / translation in VRChat and Overlays in most Streaming Applications

Whispering Tiger (Live Translate/Transcribe)

Whispering Tiger is a free and Open-Source tool that can listen/watch to any audio stream or in-game image on your machine and prints out the transcription or translation to a web browser using Websockets or over OSC (examples are Streaming-overlays or VRChat).

Content:

• Features
  • Plugins
    • List of Plugins
    • How to create Plugins
• Quickstart
• Release Downloads
• Usage
  • Usage with 3rd Party Applications
    • VRChat
    • Live Streaming Applications (OBS, vMix, XSplit ...)
    • Desktop+
• Websocket Clients
• Configurations
  • Command-line flags
  • Settings file
• Working with the Code
• FAQ
• Sources

Features

• Runs 100% locally on your machine. (Once A.I. Models are downloaded, no further internet connection is required)
• Speech recognition, translation and transcription
  • OpenAI's Whisper project, Supports ~98 languages)
• Text translation
  • LID [Language Identification] (Supports 200 languages)
  • NLLB-200 (single model, Supporting 200 languages, high accuracy)
  • M2M-100 (single model, Supporting 100 languages, high accuracy)
• OCR [Optical Character Recognition] (to capture game images and translate in-game text)
  • EasyOCR (Supports 80+ languages)
• TTS [Text-to-Speech] (Read out transcriptions/translations)
  • Silero
• VAD [Voice Activity Detection]
  • Silero-VAD
• LLM [Large language model] (Continuation of text. automatic answer generation etc.) Proof of concept
  • FLAN-T5, GPT-J, Bloomz etc. (Using the Whispering TIger Plugin for LLM)

Quickstart

For a quick and easy start, download the latest Whispering Tiger UI from here: https://github.com/Sharrnah/whispering-ui

This is a native UI application that allows keeping your Whispering Tiger version up-to-date and manage the settings more easily.

Release Downloads

Standalone Releases with all dependencies included.

Go to the GitHub Releases Page and Download from the download Link in the description or find the Latest Release here.

(because of the 2 GB Limit, no direct release files on GitHub)

• Install CUDA for GPU Acceleration (recommended)
• Extract the Files on a Drive with enough free Space.
  • (After download of medium Whisper Model + medium NLLB-200 Translation model, it can take up to 20 GB)
• Run only using the *.bat files. Edit or copy an existing start-*.bat file and edit the parameters in any text editor for your own command-line flags.
  • start-transcribe-mic.bat tries to use your default microphone and is a good starting point.

Sources

A thanks goes to

• OpenAI https://github.com/openai/whisper
• Awexander https://github.com/Awexander/audioWhisper
• Blake https://github.com/mallorbc/whisper_mic
• Meta AI (LID, NLLB-200, M2M-100) https://ai.facebook.com/blog/nllb-200-high-quality-machine-translation/
• faster-whisper https://github.com/guillaumekln/faster-whisper
• EasyOCR https://github.com/jaidedai/easyocr
• Silero (TTS, VAD) https://github.com/snakers4/silero-models