[FR] Support more input types

[yume-chan]

I want to use this library to re-mux a raw H.264 stream into a WebM file (because WebM has better support among media players than raw H.264 stream).

Because I already have an encoded stream, I don't need (or want) WebCodecs API to be involved (browser compatibility is another concern).

But currently, this library does an instanceof test against EncodedVideoChunk here:

webm-muxer/src/main.ts

Line 367 in 1e0d320

trackNumber: externalChunk instanceof EncodedVideoChunk ? VIDEO_TRACK_NUMBER : AUDIO_TRACK_NUMBER

I know I can construct EncodedVideoChunks with my encoded data, but ideally, I want to supply the buffer directly to this library, saving the extra memory allocation and copying.

I tried to modify this library like this:

diff --git a/src/main.ts b/src/main.ts
index 3109e82..840d756 100644
--- a/src/main.ts
+++ b/src/main.ts
@@ -226,7 +226,7 @@ class WebMMuxer {

 		this.writeVideoDecoderConfig(meta);

-		let internalChunk = this.createInternalChunk(chunk, timestamp);
+    let internalChunk = this.createInternalChunk(chunk, 'video', timestamp);
 		if (this.options.video.codec === 'V_VP9') this.fixVP9ColorSpace(internalChunk);

 		/**
@@ -328,12 +328,12 @@ class WebMMuxer {
 		}[this.colorSpace.matrix];
 		writeBits(chunk.data, i+0, i+3, colorSpaceID);
 	}

 	public addAudioChunk(chunk: EncodedAudioChunk, meta: EncodedAudioChunkMetadata, timestamp?: number) {
 		this.ensureNotFinalized();
 		if (!this.options.audio) throw new Error("No audio track declared.");

-		let internalChunk = this.createInternalChunk(chunk, timestamp);
+    let internalChunk = this.createInternalChunk(chunk, 'audio', timestamp);

 		// Algorithm explained in `addVideoChunk`
 		this.lastAudioTimestamp = internalChunk.timestamp;
@@ -356,7 +356,7 @@ class WebMMuxer {
 	}

 	/** Converts a read-only external chunk into an internal one for easier use. */
-	private createInternalChunk(externalChunk: EncodedVideoChunk | EncodedAudioChunk, timestamp?: number) {
+  private createInternalChunk(externalChunk: EncodedVideoChunk | EncodedAudioChunk, trackType: 'video' | 'audio', timestamp?: number) {
 		let data = new Uint8Array(externalChunk.byteLength);
 		externalChunk.copyTo(data);

@@ -364,7 +364,7 @@ class WebMMuxer {
 			data,
 			timestamp: timestamp ?? externalChunk.timestamp,
 			type: externalChunk.type,
-			trackNumber: externalChunk instanceof EncodedVideoChunk ? VIDEO_TRACK_NUMBER : AUDIO_TRACK_NUMBER
+      trackNumber: trackType === 'video' ? VIDEO_TRACK_NUMBER : AUDIO_TRACK_NUMBER
 		};

 		return internalChunk;

So I can give it plain objects. I haven't modified it to take buffers directly.

Here is my consuming code:

https://github.com/yume-chan/ya-webadb/blob/eaf3a7a3c829ebdbd4e1608c4cc0f3caf623f180/apps/demo/src/components/scrcpy/recorder.ts#L77-L100

        const sample = h264StreamToAvcSample(frame.data);
        this.muxer!.addVideoChunk(
            {
                byteLength: sample.byteLength,
                timestamp,
                type: frame.keyframe ? "key" : "delta",
                // Not used
                duration: null,
                copyTo: (destination) => {
                    // destination is a Uint8Array
                    (destination as Uint8Array).set(sample);
                },
            },
            {
                decoderConfig: this.configurationWritten
                    ? undefined
                    : {
                          // Not used
                          codec: "",
                          description: this.avcConfiguration,
                      },
            }
        );
        this.configurationWritten = true;

[Vanilagy]

First of all, note that WebM does not officially support H.264 as a codec, so I would not expect every media player to play it back correctly. That said, it's likely that many still support this codec despite it not being spec-compliant.

I do see a need for your usecase, however, of manually adding data into the muxer. I added two methods addVideoChunkRaw and addAudioChunkRaw (documented in the README), which should address this issue.

Thanks for the suggestion!

[yume-chan]

That said, it's likely that many still support this codec despite it not being spec-compliant.

Yes, even Chrome's MediaRecorder implementation also supports WebM with H.264.

A more spec-compliant solution might be using MKV instead of WebM, but I can't find a MKV muxer as good as this one.

[Vanilagy]

I mean, you could probably change the doctype in the muxer to mkv and just give your output file an .mkv extension, and it would be fine! Remember that webm is just a subset of Matroska.