[Future] support real-time translation
JackZhang1994 opened this issue · 11 comments
How to support real-time translation. Can you help provide a demo?
Hi, please check this example for real-time translation: https://github.com/ggerganov/whisper.cpp/tree/master/examples/stream
Please also check this: ggerganov/whisper.cpp#10
I want to use the offline real-time translation on Android and iOS devices. I have read stream.cpp code, but I don't find how to use it on Android and iOS devices.
Definitely, this is something I want to do too.
Streaming quality is also gated on this:
I think you could adapt the current code by starting to transcribe the audio right away and padding with silence (there's more discussion about it in ggerganov/whisper.cpp#10). So instead of waiting for the recording to complete, start sending it to Whisper right away.
I find that iOS Demo support real-time transcription. Can Android implement this in a similar way?
Another question, iOS device transcribe the audio faster than Android device. Why is that? Is there a good solution?
Thanks!
Can Android implement this in a similar way?
It should be able to.
Another question, iOS device transcribe the audio faster than Android device. Why is that? Is there a good solution?
There's two reasons I can think of at the top of my head:
- iOS uses Accelerate framework: https://developer.apple.com/accelerate/. Possibly the main repo could speed up Android ops using the neural networks API: https://developer.android.com/ndk/guides/neuralnetworks/
- Apple's CPUs are really good. Most Android devices lag far behind them in raw processing power.
Thanks for your reply!
And sorry, my English is bad.
Can Android implement this in a similar way?
It should be able to.
Do you have any plans to improve the real-time function (like the iOS demo) in the near future? I'm not good at Kotlin. And I've been trying for two days, but it still has a lot of bugs, and the transcription is very slow.
No problem at all, your English is just fine from my side ;) I'm unfortunately not able to work on it for the next few weeks but it's something I want to bring to our production apps so I'm definitely interested in this, too.
I recommend you use "tiny" for Android -- everything else will be too slow.
Also. please ensure you're using release mode when you're running from Android studio. In debug mode it will take forever. ;)
All right, thank you~