YnnkWhisperRecognizer is an add-on for Ynnk Voice Lipsync UE5 plugin. It allows to use whisper.cpp for voice recognition instead of default Vosk toolkit.
Pros: whisper is platform-independent and can be packaged for iOS, Mac, Linux (Vosk works on Windows and Android). Cons: whisper.cpp implementation doesn't work well for streamed audio input. Moreover, it's slower then Vosk.
-
Download packaged plugin (from Releases) and extract it to C:/Program Files/Epic Games/UE_(version)/Engine/Plugins/Marketplace
-
Enable the plugin in your project (Edit -> Plugins) and restart Unreal editor.
-
Download language model here: https://huggingface.co/ggerganov/whisper.cpp/tree/main You need either ggml-tiny.en.bin (English-only) or ggml-tiny.bin (multilangual). Large models work better, but for lip-sync generation rapidity and performance are more important then quallity of recognition.
-
Import .bin file to your project (drag&drop to Content Browser) as a binary archive.
-
In BeginPlay event of appropriate blueprint call (Whisper Subsystem) --> Load Model from Asset.
- And done. YnnkVoiceLipsync will use whisper now.