Opus Streaming mode
Opened this issue · 5 comments
First of all, thanks for this library.
I have a use case and don't know if it will be possible with this library.
I want to receive audio input from my Device microphone using a separate thread, encode this chunk using the library, send this encoded chunk to a server that that will decode the data using this same library. All this should happen in almost real time.
As you can see, my use case indicates that I can't have the whole data to be encoded or decoded available at once.
Now here comes my question, is this possible with this library?
Using PCM would have been easier since there is no header involved but opus was specifically chosen because of its small size and I can't change this requirement unfortunately.
Any help will be appreciated.
Thanks.
So.....the short answer is "yes", you can create an OggOpusRead/Write stream that would operate directly over a NetworkStream or something similar to send audio data between two points of the network. The code doesn't really care what kind of stream you are using unless you want to do something fancy like seeking.
There is one huge problem that you will run into: OggOpus is not suitable for real-time transmission. This is because the packetizing code will buffer "pages" at a time, up to 64Kb each, before finalizing and writing them to the output. For Opus this means that 3 to 15 seconds of audio will get stuck in a buffer before being sent out over the network. And just to clarify: Opus itself has only 5-20 ms of latency, it's the Ogg container format that adds this buffer.
There are two ways you can avoid this, and both of them involve getting away from Ogg.
First, you could send the audio data over a TCP stream as just raw Opus packets, each one with a length-prefix header. The Opus spec has a "recommended" method for how to implement this, and it would probably be the simplest for your use case. Look at section 3 "Internal Framing", as well as 3.2.1 "Frame Length Coding".
The other option would be to use RTP (Real-time protocol). This protocol was explicitly designed for your use case, and should have lower latency as it uses UDP datagrams instead of a TCP stream. It also has mechanisms for improving reliability if network connection gets lost, packets are dropped, etc. The Opus-over-RTP spec can be found here (RFC 7587). I have personally never dealt with RTP, but I assume you should be able to implement it using a generic RTP library.
First of all, thanks for your response.
I really appreciate it.
I think I really need to do some reading on this.
Here is what I planned to do originally.
Implement a callback (delegate) here https://github.com/lostromb/concentus.oggfile/blob/master/Concentus.Oggfile/OpusOggWriteStream.cs#L394 that exposes _outputStream via a ref argument.
This delegate will be called anytime "FinalizePage" is called.
via this, we can get chunks of the encoded data through the delegate.
What do you think about this procedure?
Sounds.... complicated. You want an event to fire whenever you write data to the stream, so you can intercept the data that gets written to the stream? It would be much easier to just implement your own Stream class that would process the data as it comes.
But as I said above, I don't think this would meet your real-time requirement. FinalizePage() is the code which flushes the page buffer, so by the time it gets called the data is already several seconds old
Sounds.... complicated. You want an event to fire whenever you write data to the stream, so you can intercept the data that gets written to the stream? It would be much easier to just implement your own Stream class that would process the data as it comes.
But as I said above, I don't think this would meet your real-time requirement. FinalizePage() is the code which flushes the page buffer, so by the time it gets called the data is already several seconds old
Ok, what if let's say I am willing to tolerate a few seconds delay, do you think that would work?
Sounds.... complicated. You want an event to fire whenever you write data to the stream, so you can intercept the data that gets written to the stream? It would be much easier to just implement your own Stream class that would process the data as it comes.
But as I said above, I don't think this would meet your real-time requirement. FinalizePage() is the code which flushes the page buffer, so by the time it gets called the data is already several seconds oldOk, what if let's say I am willing to tolerate a few seconds delay, do you think that would work?
Bump!!!