zhaohappy/libmedia

[Discuss] AudioContext user gesture problem

Closed this issue · 12 comments

irelance commented

When MSE not support, the player use canvas + AudioContext which will cause user promission problem.

player = new AVPlayer
// user click play button
await player.load(url)
// 4 sec pass
player.play()//throw Error: the audioContext was not started. it must be resumed after a user gesture
// A sad programmer appeared in the world

I have an idea, but uncertain whether it can be achieved.
Need a method to create an AudioContext and keep play silent audio before player.load.
the usage fake code like:

player = new AVPlayer
// user click play button
player.createAudioContextAndKeepPlaying() //use this to avoid the user gesture problem
await player.load(url)
// 4 sec pass
player.play()//keep use the audio context which use createAudioContextAndKeepPlaying
// We did it!
zhaohappy commented

Did you test it with audio file? When audioContext's state is suspended, if play video & audio file, avplayer will play video and mute audio until call resume function. If play only audio file, avplayer will throw error to keep the same behavior with mse mode. you can see code for more information. I think if you want to avoid throw error, delete that line and call AVPlayer.AudioRenderThread.fakePlay can resolve it.

irelance commented

Emmm, yes I use avplayer to play both audio file and video file, and face this problem when using IOS 15 to play audio file, replace the code not solve my problem, because audio player press the button down will load(url) first and then call play(), I have test the safari if you play audio after gesture 1 second, audioContext.state is still 'suspended'.
Video player is ok, because I call load() before user press play button.
I actually using avplayer-ui.
so, the player api play actually do is 

const preload =async (url,type="auto",autoplay=false)=>{
  await avplayer.load(url)
  switch(type){
    case "auto":
        await avplayer.play()
        if(!autoplay) {
          await avplayer.pause()
        }
        break
    default://none|metadata
  }
}
const play=async ()=>{
  await avplay.play()
  await avplayer.resume()
}

By the way, I have facing another issue of "Out of bounds memory access" on iOS device, when I play a 5MB aac music.
I try to -Oz. It can make the player play more seconds, but still OOM. It seems to make wasm smaller can avoid OOM in IOS?
For mp3 play a 15MB music still not OOM?
Do you have asm build script? I try WASM=0 and remove MAIN_MODULE and SIDE_MODULE, not success build.

irelance commented

I make a demo to prevent gesture problem:

<body>
<button>Play</button>
</body>
<script>
    const button = document.querySelector("button");
    let audioCtx;
    function initAudioCtx() {
        if (!audioCtx)audioCtx = new AudioContext();
    }

    let source

    function createAudioContextAndKeepPlaying() {
        if (!source) {
            source = audioCtx.createBufferSource();
            source.connect(audioCtx.destination);
            source.start();
            audioCtx.suspend();//to avoid difference between firefox chrome and safari
        }
    }

    //simulate load music from server
    async function load() {
        await new Promise(resolve => setTimeout(resolve, 3000))

        function createBuffer(audioCtx) {
            const length =  audioCtx.sampleRate*1
            const buffer = new AudioBuffer({
                numberOfChannels: 1,
                length,
                sampleRate: audioCtx.sampleRate,
            });
            const nowBuffering = buffer.getChannelData(0);
            for (let i = 0; i < length; i++) {
                nowBuffering[i] = Math.random() * 2 - 1;
            }
            return buffer
        }

        source.buffer = createBuffer(audioCtx)
        console.log('audioCtx.currentTime',audioCtx.currentTime)
    }
    async function play() {
        audioCtx.resume()//to avoid difference between firefox chrome and safari
    }

    button.onclick = async () => {
        initAudioCtx()
        createAudioContextAndKeepPlaying()//just make it pass the user gesture
        await load()
        await play()
    };
</script>
zhaohappy commented

It looks like on iOS 15, we create audioContext within a short period of time after gesture will get running state, otherwise will get suspended state. You can try the following code to test whether it can be solve:

const player = new AVPlayer();
button.onclick = async () => {
       // immediately create audioContext after gesture
       if (!AVPlayer.audioContext) {
            await AVPlayer.startAudioPipeline()
       }
        await player.load()
        await player.play()
    };
zhaohappy commented

By the way, I have facing another issue of "Out of bounds memory access" on iOS device, when I play a 5MB aac music. I try to -Oz. It can make the player play more seconds, but still OOM. It seems to make wasm smaller can avoid OOM in IOS? For mp3 play a 15MB music still not OOM? Do you have asm build script? I try WASM=0 and remove MAIN_MODULE and SIDE_MODULE, not success build.

"Out of bounds memory access" may also be a memory access error because of pointer exception, the aac decoder wasm you used is compiled by yourself from lower version of ffmpeg? libmedia cannot support asm.js, so I don't have asm build script.

irelance commented

No, I use FFmpeg 7.0 to compile. And use your baseline version to test as well.

zhaohappy commented

Can this be reproduced on desktop chrome? Can you upload your file so I can debug it.

irelance commented

desktop is ok. it just happens on iOS. the music is Trans by ffmpeg cli
test.zip

zhaohappy commented

I don't have an iOS 15 device, so I used the simulator to reproduced it. For AudioContext user gesture problem, I
test the following code is ok:

const player = new AVPlayer();
button.onclick = async () => {
       // immediately create audioContext after gesture
       if (!AVPlayer.audioContext) {
            // don't wait call startAudioPipeline
            AVPlayer.startAudioPipeline()
            // create a BufferSource
           AVPlayer.audioContext.createBufferSource()
       }
        await player.load()
        await player.play()
    };

I think it is ios15's bug, and cannot reproduced in ios17. In addition, I have not facing issue of "Out of bounds memory access", can you upload your error stack and the result of execute CHeap.Allocator.inspect() in Console when error occur.

irelance commented

This test use commit 4e9df50 wasm

photos

Uncaught (in promise)  {"name": "RuntimeError", "message": "Out of bounds memory access (evaluating 'this.decoder.call("decoder_decode",t)')", "stack": "<?>.wasm-function[545]@[wasm code]\n<?>.wasm-function[36]@[wasm code]\nwasm-stub@[native code]\ndecode@http://192.168.137.1:9000/js/video/avplayer/avplayer.js:1:92029\n@http://192.168.137.1:9000/js/video/avplayer/avplayer.js:1:196791\nasyncFunctionResume@[native code]\n@[native code]\npromiseReactionJobWithoutPromise@[native code]\npromiseReactionJob@[native code]"}

where avplayer.js:1:92029
is commit 4e9df50fb4d3b16e560ea0c035a256c5e8518508 dist/avplayer-ui/avplayer.js:1:102884 e=this.decoder.call

where avplayer.js:1:196791
is commit 4e9df50fb4d3b16e560ea0c035a256c5e8518508 dist/avplayer-ui/avplayer.js:1:207470 e=h.decoder.decode

[src\cheap\allocator\AllocatorJS.ts][line 624] [error] Got invalid sized chunk at 2097872 (3261027220)
zhaohappy commented

This looks like a memory leak but I haven't been able to reproduce. I need more information, the specific version of Safari and iOS, it is used ShareArraubuffer? or has specific steps to reproduce? Thanks.

irelance commented

iOS 15.8.3

User Agent
Mozilla/5.0 (iPhone; CPU iPhone OS 15_8_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6.6 Mobile/15E148 Safari/604.1

SharedArrayBuffer is undefined

If there is no solution, let it go. Time will make old iPhones outdate.