Vanilagy/webm-muxer

No sound in final output when muxing audio and video. Also duration metadata is missing.

akashdeep000 opened this issue · 2 comments

akashdeep000 commented

Hi @Vanilagy

First of all thanks for this amazing library.
I want mux two separate audio and video files that will be downloaded from a remote server.

Current problems that I'm facing:

  1. When passing only the video file the output file does not have duration metadata and I am unable to seek the video.
  2. When passing both the audio and video file output file problem one exits and the output video's size increases but when playing the video there is no audio.

Here are the files that I'm using for the test:

Here is the function that I am currently using:

const muxAndDownload = async () => {
    const {
        readable,
        writable
    } = new TransformStream()
    const writer = writable.getWriter();

    let muxer = new Muxer({
        target: new StreamTarget({
            onData: (data) => {
                console.log(data)
                writer.write(data)
            }
        }),
        video: {
            codec: 'V_VP9',
            width: 256,
            height: 144,
            frameRate: 25,
        },
        audio: {
            codec: "A_OPUS",
            numberOfChannels: 2,
            sampleRate: 48000,
        },
    });

    const videoRes = await fetch("https://pub-951fa3482dca41f6bf9fa25a7953175d.r2.dev/yt144.webm")
    const audioRes = await fetch("https://pub-951fa3482dca41f6bf9fa25a7953175d.r2.dev/ytaudio.webm")

    const video = await videoRes.arrayBuffer()
    const audio = await audioRes.arrayBuffer()

    muxer.addVideoChunkRaw(new Uint8Array(video), "key", 0)
    muxer.addAudioChunkRaw(new Uint8Array(audio), "key", 0)
    muxer.finalize()
    // downloading the final muxed video stream using "streamsaver" package.
    readable.pipeTo(streamSaver.createWriteStream('media.webm'))
    writer.close()
}

Add raw video chunk part by part rather than waiting for full download (If possible):

Also If possible could you help with how I can add video and audio data to the muxer chunk by chunk rather than waiting for the whole video and audio files?

As the muxer.addVideoChunkRaw function needs the timestamp of the video chunk, how could I get that when downloading the video file from the remote server? (I know the total video and audio duration if that can help in any way).

Vanilagy commented

Hi! You're using this library incorrectly.

Firstly, StreamTarget's onData callback, as specified in the README, takes in two arguments, data and position. You NEED to respect position! More concretely, you're expected to write data at the offset position. So, it is not an append-only buffer. If you insist on using streams, you'll need to set streaming to true (also in the README). But this removes certain capabilities.

Secondly, you can't just add full video files as chunks as input. The chunks should be raw codec data! You're passing in entire media files. So, if you want to combine two media files, you need to first demux the two incoming files, extract out the chunks, then mux them together again. This library doesn't help with that as it does not implement a demuxer.

Vanilagy commented

I'm closing this issue for now. Reopen it if you feel there's a need for it :)