FLAC__stream_encoder_process_interleaved errors out inconsistently
Closed this issue · 17 comments
I am using libflac4-1.3.2 and getting errors quite frequently , especially when recording and encoding audio on Firefox. All times, the errors are in
FLAC__stream_encoder_process_interleaved
Above returns false. Calling FLAC__stream_encoder_get_state() afterwards returns 0.
Also, in some instances I am getting script terminated by timeout
Error: Script terminated by timeout at:
we@http://localhost:8080/js/test.js:191:27939
dn@http://localhost:8080/js/test.js:198:20290
cn@http://localhost:8080/js/test.js:198:8495
Qn@http://localhost:8080/js/test.js:200:8112
a@http://localhost:8080/js/test.js:188:1127
FLAC__stream_encoder_process_interleaved@http://localhost:8080/js/test.js:204:10929
Any clues.
[FYI: I also saw the script terminated by timeout errors on the demo https://mmig.github.io/speech-to-flac/ page]
could give some more details/context?
Does this occur regardless of how large the to-be-encoded data is?
(e.g. if from microphone/recording, after what duration roughly does this happen, or is there no clear pattern?)
Do you have the development-tools/-console of Firefox open?
(or does this happen regardless if the console is opened or closed)
Do you encode a stream ("bit by bit") or a complete blob ("all at once")?
(if you can, would it be possible to provide sample input for which this error occurs?)
Do you use libflac4-1.3.2 as-is, or is it somehow recompiled by a 3rd party build-tool (e.g. minimized or otherwise "optimized")?
Did you try other variants, e.g. the wasm variants, and/or more optimized-builds (i.e. min variants)?
Does this occur regardless of how large the to-be-encoded data is?
Yes. I have not tried at all with larger data, so cant say. All my tests have been 10-45 seconds long.
(e.g. if from microphone/recording, after what duration roughly does this happen, or is there no clear pattern?)
Realised, this works fine if I just do one recording. And error caused only when I was calling Flac.init_encoder_stream the second (or subsequent) times.
Do you have the development-tools/-console of Firefox open?
Yes. Will try again with console closed.
Do you encode a stream ("bit by bit") or a complete blob ("all at once")?
I am doing bit by bit, with buffer size 4096. Will try again with other sizes.
Do you use libflac4-1.3.2 as-is, or is it somehow recompiled by a 3rd party build-tool (e.g. minimized or otherwise "optimized")?
I am using gulp (libsass) to include this js into my js.
Did you try other variants, e.g. the wasm variants, and/or more optimized-builds (i.e. min variants)?
No
Thanks for looking into it. I will try some more variations as per your suggestions in order to narrow it.
Realised, this works fine if I just do one recording. And error caused only when I was calling Flac.init_encoder_stream the second (or subsequent) times.
do you create (and use) a new encoder before calling Flac.init_encoder_stream?
i.e. something like
var aNewEncoder = Flac.create_libflac_encoder(...
var status = Flac.init_encoder_stream(aNewEncoder , or do you try to reuse the the old/previous encoder?
I create a new encoder every time. ie
// Round 1
var flac_encoder = Flac.create_libflac_encoder(...
var status = Flac.init_encoder_stream(flac_encoder, ..
...
// clear
flac_encoder = null;
// Round 2
flac_encoder = Flac.create_libflac_encoder(...
...
@russaa I am not sure if it is the same error as @supratims is experiencing, but here is a reproducible bunch of errors with encoding:
Here is an example with libflac4-1.3.2.dev.wasm.js; non-dev and non-wasm versions have similar behavior. Host is Chromium 71.0.3578.98.
const encoder_id = Flac.create_libflac_encoder(44100, 1, 16, 5, 0, true, 0);
> undefined
cb = (data) => { console.log(data); };
> (data) => { console.log(data); }
Flac.init_encoder_stream(encoder_id, cb, cb);
> VM3703:1
> Uint8Array(4) [102, 76, 97, 67]
> VM3703:1
> Uint8Array(38) [0, 0, 0, 34, 16, 0, 16, 0, 0, 0, 0, 0, 0, 0, 10, 196, 64, 240, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
> VM3703:1
> Uint8Array(44) [132, 0, 0, 40, 32, 0, 0, 0, 114, 101, 102, 101, 114, 101, 110, 99, 101, 32, 108, 105, 98, 70, 76, 65, 67, 32, 49, 46, 51, 46, 50, 32, 50, 48, 49, 55, 48, 49, 48, 49, 0, 0, 0, 0]
> 0
>
Flac.FLAC__stream_encoder_get_state(encoder_id);
> 0
buf = Int32Array.from("1234".repeat(1000))
> Int32Array(4000) [1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, …]
Flac.FLAC__stream_encoder_process_interleaved(encoder_id, buf.buffer, buf.length);
>1
Flac.FLAC__stream_encoder_get_state(encoder_id);
>0
Flac.FLAC__stream_encoder_process_interleaved(encoder_id, buf.buffer, buf.length);
> VM3703:1
> Uint8Array(715) [255, 248, 201, 8, 0, 149, 16, 21, 33, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 9, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, 66, 16, 132, 33, 8, …]
> 1
>
Flac.FLAC__stream_encoder_get_state(encoder_id);
>0
Flac.FLAC__stream_encoder_process_interleaved(encoder_id, buf.buffer, buf.length);
> VM3703:1
> Uint8Array(4347) [255, 248, 201, 8, 1, 146, 16, 23, 160, 0, 64, 0, 128, 1, 0, 2, 0, 4, 0, 8, 0, 16, 0, 32, 0, 64, 0, 128, 1, 0, 2, 0, 4, 0, 8, 0, 16, 0, 32, 0, 64, 0, 128, 1, 0, 2, 0, 4, 0, 8, 0, 16, 0, 32, 0, 64, 0, 128, 1, 0, 2, 0, 4, 0, 8, 0, 16, 0, 32, 0, 64, 0, 128, 1, 0, 2, 0, 4, 0, 8, 0, 16, 0, 32, 0, 64, 0, 128, 1, 0, 2, 0, 4, 0, 8, 0, 16, 0, 32, 0, …]
>
>1
Flac.FLAC__stream_encoder_get_state(encoder_id);
>0
Flac.FLAC__stream_encoder_process_interleaved(encoder_id, buf.buffer, buf.length);
>0 // <==== this signals an error! ("false")
Flac.FLAC__stream_encoder_get_state(encoder_id);
>4 // <=== error state, verification failed
Flac.FLAC__stream_encoder_process_interleaved(encoder_id, buf.buffer, buf.length);
> VM3703:1
> Uint8Array(8201) [255, 248, 201, 8, 2, 155, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …]
> 1
>
Here the very last return code from FLAC__stream_encoder_process_interleaved was 1 (true), but it's easy to see that the returned data does not look like the expected FLAC frame (we got an array of 8201 values, mostly zeroes)
Also, the call to FLAC__stream_encoder_process_interleaved just before it returned 0.
Flac.FLAC__stream_encoder_get_state(encoder_id);
>4
And FLAC's own validation of the encoded stream now fails with 4 == FLAC__STREAM_ENCODER_VERIFY_MISMATCH_IN_AUDIO_DATA, https://xiph.org/flac/api/group__flac__stream__encoder.html#ggac5e9db4fc32ca2fa74abd9c8a87c02a5a011e3d8b2d02a940bfd0e59c05cf5ae0
If we try to feed more data, we will always get FLAC__STREAM_ENCODER_VERIFY_MISMATCH_IN_AUDIO_DATA, and the same strange output frames of size 8201.
Moreover, attempts to finish/cleanup the encoder yield memory errors:
Flac.FLAC__stream_encoder_finish(encoder_id);
> (unknown) Uncaught RuntimeError: memory access out of bounds
> at _free (wasm-function[273]:68)
> at _FLAC__MD5Final (wasm-function[131]:591)
> at _FLAC__stream_decoder_finish (wasm-function[148]:140)
> at _FLAC__stream_encoder_finish (wasm-function[208]:988)
> at Module._FLAC__stream_encoder_finish (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:5669:56)
> at ccall (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:568:18)
> at Object.FLAC__stream_encoder_finish (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:576:12)
> at <anonymous>:1:6
>
Flac.FLAC__stream_encoder_delete(encoder_id);
> libflac4-1.3.2.dev.wasm.js:6079 Uncaught abort() at Error
> at jsStackTrace (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:1093:13)
> at stackTrace (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:1110:12)
> at Object.abort (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:6073:44)
> at _abort (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:5138:22)
> at _free (wasm-function[273]:1701)
> at _free_ (wasm-function[211]:2668)
> at _FLAC__stream_encoder_finish (wasm-function[208]:1224)
> at _FLAC__stream_encoder_delete (wasm-function[207]:91)
> at Module._FLAC__stream_encoder_delete (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:5665:56)
> at Object.ccall (http://localhost:8080/libflac4-1.3.2.dev.wasm.js:568:18)
Disabling validation causes the error messages to go away, but the output "FLAC" is still the same Uint8Array(8201) with short non-zero prefix, all else being zeros.
My guess is that I'm doing something fundamentally wrong -- the library effectively doesn't work for me at all; but currently I cannot see where exactly the problem lies.
Just FTR, rebuilding libflacjs from source didn't help (the Makefile needed to be slightly amended to work with the current Emscripten, so it's not the 100% fair reproduction).
@ilya-edrenkin I think the problem may be that you pass in the Int32Array's buffer instead of the typed array itself, i.e. try passing in buf instead of buf.buffer:
Flac.FLAC__stream_encoder_process_interleaved(encoder_id, buf, buf.length)@russaa you are right indeed! Thanks, I spent a few hours looking in a wrong place.
The reason I supplied this argument in this (incorrect) way is that the proper way didn't work for me at all in some other context. I was calling libflacjs via js from some other webassembly module, and the underlying ArrayBuffer was this whole wasm module's memory (viewed as typed array with correct byte offset and length).
However if we look at the emscripten-generated code, it's clear why this caused problems:
FLAC__stream_encoder_process_interleaved: function(encoder, buffer, num_of_samples){
// get the length of the data in bytes
var numBytes = buffer.length * buffer.BYTES_PER_ELEMENT;
// console.log("DEBUG numBytes: " + numBytes);
// malloc enough space for the data
var ptr = Module._malloc(numBytes);
// get a bytes-wise view on the newly allocated buffer
var heapBytes= new Uint8Array(Module.HEAPU8.buffer, ptr, numBytes);
// console.log("DEBUG heapBytes: " + heapBytes);
// copy data into heapBytes
heapBytes.set(new Uint8Array(buffer.buffer));
var status = Module.ccall('FLAC__stream_encoder_process_interleaved', 'number',
['number', 'number', 'number'],
[encoder, heapBytes.byteOffset, num_of_samples]
);
Module._free(ptr);
return status;
},
The culprit is heapBytes.set(new Uint8Array(buffer.buffer)); which attempts to copy the whole underlying buffer.buffer (in this case the whole calling module's memory, which of course does not fit into heapBytes). If I'm not mistaken, heapBytes.set(new Uint8Array(buffer)); would have done the right thing, copying the typed array.
So it only works in special cases when the underlying ArrayBuffer has offset 0 and length equal to the typed array's view byte length.
Would you agree that this looks like an emscripten bug?
yes, it's part of the wrapper code for libflac -- good catch
I'll make a bugfix later & publish it later
Could you test it again then, with circumstances that failed you before?
Sure, I tested it again and it works now.
Re the fix: please see the pull request, #13
The relevant line is this one https://github.com/mmig/libflac.js/pull/13/files#diff-3feb20c59b0cbbc9214c31a112e7008aR1034, but to make it build I also had to slightly edit the Makefile, see the PR for details.
I guess the OGG build is currently broken (I would expect at least to see building of libflac with OGG enabled as a dependency), but that's probably out of scope of this fix.
sorry I was working in parallel and did not see your PR sooner -- could you merge the changes and push them into your PR?
(you can ignore/overwrite or whatever, any changes in /dist, I will recompile again after integrating the PR)
@russaa no problem! I would rather close my PR now as there is nothing substantial left -- just the Makefile tweaks, which you can reuse if you wish so. But I am not 100% sure that what works in my emscripten environment would work for you (as vice versa it doesn't work), so it would be probably better if you decide on this yourself :)
Thank you!
OK, good -- and thanks for your work on tracking this error
@russaa Is there any new release you are planning with all the recent fixes you made ?
the latest changes/updates (including the ones for this issue) are contained in the master branch (with version 4.0.1)