superpoweredSDK/web-audio-javascript-webassembly-SDK-interactive-audio

JS WASM can't process more than 12 AdvancedAudioPlayers

tralves opened this issue · 20 comments

tralves commented

Good day,

I noticed that when processing more than 12 AdvancedAudioPlayers in WASM, some players stop playing and eventually nothing plays. This does not happen on the native platforms I tested (Android, iOS and OSX).

I did extensive testing in these sample apps:

I know that:

  • When it happens, processStereo stops creating an output and returns false, and getPositionMs doesn't advance.
  • With:
    • 10 simultaneous AAPs work fine,
    • 20 AAPs, sound stops after 9515 processStereo()
    • 30 APPs, sound stops after 3695 processStereo()
    • 40 AAPs, sound stops after 151 processStereo()
  • It's not a performance issue. SP takes about 100ms to process over 1000s of audio;
  • It's not a timing issue. I tested processing the audio outside the process() callbacks. Even with all the time in the world, it fails to process all the players;
  • Not a memory issue. Doesn't matter if the WASM is given 32mb or 256mb;
  • It's about the number of playing AAPlayers that are in the playing state at a given time. I can instantiate 40 AAPlayers but if it only plays 5 of them, everything goes fine.

I hope this helps and it's an easy fix. I am out of ideas. Tell me if you want me to try something.
Tiago

gaborszanto commented

Is this with our JavaScript API or the (deprecated, experimental) bitcode version?

tralves commented
gaborszanto commented

Our JavaScript API grows memory dynamically and reports the allocations on the console. How big is the allocation when this happens?

tralves commented

After loading all the samples, this is the memory report:

WASM memory 1257903821594: 512 kb stack, 34 mb heap, 35 mb total.

When playing, I don't get any more memory growth reposts. It's also important to notice that if I create the same 20 Players (same memory allocated) but only ever processStereo() on 5 of them, the problem doesn't happen. That's why I guess is that it's not memory related.

gaborszanto commented

I just checked the source of https://lucid-newton-d41384.netlify.app/. I see the same AudioInMemory format is loaded into multiple players. We didn't think about this use-case. Please try a separate sample for each player, let's see if that's the problem.

tralves commented

I had already tested that, but just to make sure, I updated the https://lucid-newton-d41384.netlify.app/ app. With 20 samples, the sound stops. With 10 samples (just commenting out samples from const sampleUrls[], it works;

gaborszanto commented

playSynchronizedToPosition is meant to be used for a different use-case. What happens if you simply call play()?

tralves commented

I had tried that before and tried again. Same outcome. With 20 samples it stops playing, 10 samples works fine.

gaborszanto commented

How much time does it take for processing 20 players in the audio processing callback?

gaborszanto commented

How much time does it take for processing 20 players in the audio processing callback?

tralves commented
gaborszanto commented

Measurement on the main thread is impossible, it should be done directly in the audio processing callback. Web audio typically runs with the buffer size of 128, which is only 2.666 ms if the sample rate is 48000 Hz.

If the performance API is not available in the Audio Worklet, then for the measurement purposes please switch to ScriptProcessorNode by changing superpoweredwebaudio.js. Your processing will run in the main scope and you can properly measure how much percentage is spent with audio processing in the audio processing callback.

tralves commented
gaborszanto commented

That kind of measurement is not usable. Please measure how I described above.

tralves commented

Here goes the processing results of 20 samples until the sound stops:

WASM memory 1431559195757: 512 kb stack, 82 mb heap, 84 mb total.
processAudio false: 2.100000001490116
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio true: 3.899999998509884
2processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.29999999701976776
3processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.5999999940395355
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.09999999403953552
2processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.3999999985098839
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.19999999552965164
2processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.3999999985098839
processor.js:70 processAudio true: 0.4000000059604645
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.3999999985098839
processor.js:70 processAudio true: 0.09999999403953552
processor.js:70 processAudio true: 0.4000000059604645
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.6999999955296516
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.3999999985098839
2processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.29999999701976776
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.5
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.19999999552965164
processor.js:70 processAudio true: 0.30000000447034836
processor.js:70 processAudio true: 0.20000000298023224
processor.js:70 processAudio true: 0.10000000149011612
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.19999999552965164
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0.20000000298023224
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.20000000298023224
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.10000000149011612
processor.js:70 processAudio false: 0
processor.js:70 processAudio false: 0.19999999552965164

These times are measured in ms, so the speed is more than enough. Something else is causing the processing to stop when there are these many players.

gaborszanto commented

So this was all measured with ScriptProcessorNode, and which API provided the times exactly?

tralves commented

Ah, missed your question.
Yes, it is using ScriptProcessorNode.
Each processAudio consists of processStereo() on 20 players. Players started with simple player.play();.

This is the commit for this test: Casperaki/example_advancedaudioplayer_perf@9aa9cb6

Btw, with fewer players, times were practically the same. There is something weird when playing lots of players.

gaborszanto commented

Please add some code that checks the contents of outputBuffer.array[0...buffersize * 2] for not a number values, after each player.processStereo. Maybe one of the players goes rogue and puts invalid audio into the buffer.

hugorodrigues commented

I was able to reproduce this issue while opening #32, very similar behaviour.

gaborszanto commented

Discovered a memory allocation bug. Please update to the latest version and check if it fixes it.