Move whisper to Barracuda (Sentis)
Closed this issue · 3 comments
Unity has Barracuda framework for ONNX models inference. Main benefit is that it uses CPU and GPU inference on all Unity supported platforms.
While whisper.cpp is nice, it need to be compiled from C++ for every platform. Right now it supports GPU only for CUDA which is annoying to instal. Barracuda GPU supports almost all platforms that supports compute shaders.
Biggest downside is that I would need to rewrite all pre-post processing of whisper from C++ to C#. It's 5k+ lines of code of whisper.cpp. Doable, but will take a lot of work.
It also interesting to test if Barracuda will even give any performance boost before starting this project.
Unity supports native plug-ins, which are libraries of native code you can write in languages such as C, C++, and Objective-C. Plug-ins allow the code you write in C# to call functions from these libraries. This feature allows Unity to integrate with middleware libraries or existing C/C++ code.
So, maybe it is possible to keep the C++ code for pre- and post-processing and just call it from C#?
So, maybe it is possible to keep the C++ code for pre- and post-processing and just call it from C#?
This will require to change original whisper.cpp code and maintain own fork of it. It also means that you would need to constantly transfer data between managed C# and unmanaged C++.
While it possible, I don't think it worth it. Will need to research more how whisper.cpp and its internal state manegment works.
Recently, Hugging Face and Unity team has released their implementation of whisper inference using Sentis (previously known as Barracuda). Check this article for more info.
From a quick tests of whisper.tiny model, whisper.cpp seems to be marginally faster. However, it doesn't have external dependencies like natively compiled libs or CUDA Toolkit.
Rewriting current inference to Sentis will drop all existing code and whisper.cpp binaries. I think, it will be out of the scope of this project. At least for now.