Safari 17 brings MSE to iOS

Question

Safari 17 brings MSE to iOS

alexg-axis opened this issue a year ago · 10 comments

Safari 17 on macOS and iOS brings a new ManagedMediaSource implementation that Apple claim is backwards compatible with MediaSource. It does come with some nice features on its own, but more importantly, this should mean that we can finally get DASH working in-browser on iOS. In theory at least.

I've tried to make use of this new feature in rx-player by simply modifying src/compat/browser_compatibility_types.ts.

 interface WindowsWithMediaSourceImplems extends Window {
   MediaSource? : typeof MediaSource;
+  ManagedMediaSource? : typeof MediaSource;
   MozMediaSource? : typeof MediaSource;
   WebKitMediaSource? : typeof MediaSource;
   MSMediaSource? : typeof MediaSource;
 }

 const MediaSource_ : typeof MediaSource | undefined =
   win === undefined                            ? undefined :
+   !isNullOrUndefined(win.ManagedMediaSource)   ? win.ManagedMediaSource :
   !isNullOrUndefined(win.MediaSource)          ? win.MediaSource :
   !isNullOrUndefined(win.MozMediaSource)       ? win.MozMediaSource :
   !isNullOrUndefined(win.WebKitMediaSource)    ? win.WebKitMediaSource :
                                                  win.MSMediaSource;

(Actually, I just set MediaSource_ directly to ManagedMediaSource to ensure it was used on macOS when testing, even though MediaSource was available).

Unfortunately, this doesn't seem to work as intended. On macOS, nothing really happens.

macOS logs when loading the page

[Log] HTMLMediaElement::HTMLMediaElement(7B1EDB1FD7E3FBDC) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(A39B97CE245F78A5) 
[Log] HTMLMediaElement::insertedIntoAncestor(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::didFinishInsertingNode(7B1EDB1FD7E3FBDC) 
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequireUserGestureForFullscreen
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequirePageConsentToLoadMedia
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequireUserGestureToAutoplayToExternalDevice
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequireUserGestureToControlControlsManager
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequirePlaybackToControlControlsManager
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding InvisibleAutoplayNotPermitted
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequireUserGestureForAudioRateChange
[Info] MediaElementSession::addBehaviorRestriction(7B1EDB1FD7E3FBDC) adding RequireUserGestureToShowPlaybackTargetPicker
[Log] MediaSessionCoordinator::MediaSessionCoordinator(64871A93) 
[Log] MediaSessionCoordinator::setMediaSession(64871A93) 
[Log] MediaSession::MediaSession(7FA3EBD0) 
[Log] MediaElementSession::clientWillBeginAutoplaying(7B1EDB1FD7E3FBDC) state = Idle
[Log] MediaElementSession::setState(7B1EDB1FD7E3FBDC) Autoplaying
[Log] MediaSessionManagerCocoa::sessionCanProduceAudioChanged(0) 
[Log] MediaSessionManagerCocoa::sessionCanProduceAudioChanged(0) 
[Log] HTMLMediaElement::insertedIntoAncestor(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::didFinishInsertingNode(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::insertedIntoAncestor(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::didFinishInsertingNode(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::insertedIntoAncestor(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::didFinishInsertingNode(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::insertedIntoAncestor(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::didFinishInsertingNode(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::insertedIntoAncestor(7B1EDB1FD7E3FBDC) 
[Log] HTMLMediaElement::didFinishInsertingNode(7B1EDB1FD7E3FBDC) 
[Log] MediaSessionManagerCocoa::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(0), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManagerCocoa::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Log] MediaSessionManagerCocoa::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(0), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManagerCocoa::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Info] MediaElementSession::clientDataBufferingTimerFired(7B1EDB1FD7E3FBDC) visible = false

macOS logs when playing tears of steal

[Info] API: Calling loadvideo – "https://www.bok.net/dash/tears_of_steel/cleartext/stream.mpd" – "dash"
[Info] DRM: Clearing-up DRM session.
[Info] DRM: Nothing to clear. Returning right away. No state = – true
[Info] API: playerStateChange event – "LOADING"
[Info] Init: Clearing HTMLMediaElement's src
[Info] Init: Creating MediaSource
[Info] Init: Attaching MediaSource URL to the media element – "blob:https://localhost:8443/24302411-086e-4bb5-bdfd-79cc5a2c8a70"
[Log] HTMLMediaElement::setPreload(9D9CBDD4B0176ED4) auto
[Log] HTMLMediaElement::HTMLMediaElement(313B08342303D0D4) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(7FD5C0A2937FD99E) 
[Log] HTMLMediaElement::HTMLMediaElement(1F3EF0E2CD76E75) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(5448BDED2BF68018) 
[Log] HTMLMediaElement::HTMLMediaElement(BB487811C8B8FF0E) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(755D8802679159F2) 
[Log] HTMLMediaElement::HTMLMediaElement(7BCD0270E62EBC86) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(4094B878987C9182) 
[Log] HTMLMediaElement::HTMLMediaElement(8D9C7B8BA585FB2C) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(AC26823AE727CF73) 
[Log] HTMLMediaElement::HTMLMediaElement(A3B5564EF9DA1745) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(18A07C72B59B2DC7) 
[Log] HTMLMediaElement::HTMLMediaElement(C0A6AB0F6E6EAFA2) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(A8AE017805E9C67E) 
[Log] HTMLMediaElement::HTMLMediaElement(EC638B23E6D3782) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(6BD5285EA384D23F) 
[Log] HTMLMediaElement::HTMLMediaElement(566BAE9588A77BA3) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(4F02D4F1617D8ACB) 
[Log] HTMLMediaElement::HTMLMediaElement(6078D10DC99A188C) 
[Log] HTMLVideoElementPictureInPicture::HTMLVideoElementPictureInPicture(D84EE1D8E32AF46C) 
[Log] HTMLMediaElement::prepareForLoad(9D9CBDD4B0176ED4) gesture = true
[Info] MediaElementSession::removeBehaviorRestriction(9D9CBDD4B0176ED4) removed RequireUserGestureForAudioRateChange, RequireUserGestureForFullscreen, RequireUserGestureToShowPlaybackTargetPicker, RequireUserGestureToAutoplayToExternalDevice, InvisibleAutoplayNotPermitted, RequireUserGestureToControlControlsManager
[Log] HTMLMediaElement::createMediaPlayer(9D9CBDD4B0176ED4) 
[Log] MediaSessionManagerCocoa::addSession(0) (9D9CBDD4B0176ED4)
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::cancelPendingEventsAndCallbacks(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::setPlaybackRate(9D9CBDD4B0176ED4) 1
[Log] MediaElementSession::clientWillBeginAutoplaying(9D9CBDD4B0176ED4) state = Autoplaying
[Log] HTMLMediaElement::setShouldDelayLoadEvent(9D9CBDD4B0176ED4) true
[Info] MediaElementSession::removeBehaviorRestriction(9D9CBDD4B0176ED4) removed RequirePageConsentToLoadMedia
[Debug] MediaSource::addedToRegistry(0) 
[Log] HTMLMediaElement::prepareForLoad(9D9CBDD4B0176ED4) gesture = true
[Log] HTMLMediaElement::createMediaPlayer(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::cancelPendingEventsAndCallbacks(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::setPlaybackRate(9D9CBDD4B0176ED4) 1
[Log] MediaElementSession::clientWillBeginAutoplaying(9D9CBDD4B0176ED4) state = Autoplaying
[Warning] DASH: No usable id found in the Period. Generating one.
[Info] MF: Manifest parsed in 9.99999999999909ms
[Debug] HTMLMediaElement::dispatchEvent(9D9CBDD4B0176ED4) emptied
[Log] HTMLMediaElement::selectMediaResource(9D9CBDD4B0176ED4) lambda(), task fired
[Info] HTMLMediaElement::loadResource(9D9CBDD4B0176ED4) [url] – {containerType: "", codecs: "codecs", profiles: "profiles"}
[Info] HTMLMediaElement::loadResource(9D9CBDD4B0176ED4) m_currentSrc is [url]
[Log] HTMLMediaElement::loadResource(9D9CBDD4B0176ED4) loading MSE blob
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerEngineUpdated(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(9D9CBDD4B0176ED4) task scheduled
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::selectMediaResource(9D9CBDD4B0176ED4) using 'src' attribute url
[Info] MediaElementSession::canShowControlsManager(9D9CBDD4B0176ED4) returning FALSE: no audio
[Log] MediaSessionManagerCocoa::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(1), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManagerCocoa::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Log] MediaSessionManagerCocoa::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(1), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManagerCocoa::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Debug] HTMLMediaElement::dispatchEvent(9D9CBDD4B0176ED4) loadstart
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(9D9CBDD4B0176ED4) lambda(), task fired
[Log] HTMLMediaElement::mediaEngineWasUpdated(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] MediaElementSession::mediaEngineUpdated(9D9CBDD4B0176ED4) 
[Log] VideoLayerManagerObjC::setVideoFullscreenFrame(9D9CBDD4B0176ED4) 0, 0, 0, 0
[Info] HTMLMediaElement::scheduleUpdateMediaState(9D9CBDD4B0176ED4) task scheduled
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Log] VideoLayerManagerObjC::didDestroyVideoLayer(9D9CBDD4B0176ED4) 
[Log] VideoLayerManagerObjC::setTextTrackRepresentationLayer(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerEngineUpdated(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(9D9CBDD4B0176ED4) task scheduled
[Log] MediaSource::setLogIdentifier(9D9CBDD4B0176ED4) 
[Debug] MediaSource::setPrivateAndOpen(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::setShouldDelayLoadEvent(9D9CBDD4B0176ED4) false
[Log] MediaSource::onReadyStateChange(9D9CBDD4B0176ED4) old state = closed, new state = open
[Info] HTMLMediaElement::scheduleUpdateMediaState(9D9CBDD4B0176ED4) lambda(), task fired
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(9D9CBDD4B0176ED4) lambda(), task fired
[Log] HTMLMediaElement::mediaEngineWasUpdated(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] MediaElementSession::mediaEngineUpdated(9D9CBDD4B0176ED4) 
[Log] VideoLayerManagerObjC::setVideoFullscreenFrame(9D9CBDD4B0176ED4) 0, 0, 0, 0
[Info] HTMLMediaElement::scheduleUpdateMediaState(9D9CBDD4B0176ED4) task scheduled
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerEngineUpdated(9D9CBDD4B0176ED4) AVFoundation MediaSource Engine
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(9D9CBDD4B0176ED4) task scheduled
[Info] HTMLMediaElement::scheduleUpdateMediaState(9D9CBDD4B0176ED4) lambda(), task fired
[Log] HTMLMediaElement::scheduleMediaEngineWasUpdated(9D9CBDD4B0176ED4) lambda(), task fired
[Log] HTMLMediaElement::mediaEngineWasUpdated(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] MediaElementSession::mediaEngineUpdated(9D9CBDD4B0176ED4) 
[Log] VideoLayerManagerObjC::setVideoFullscreenFrame(9D9CBDD4B0176ED4) 0, 0, 0, 0
[Info] HTMLMediaElement::scheduleUpdateMediaState(9D9CBDD4B0176ED4) task scheduled
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::scheduleUpdateMediaState(9D9CBDD4B0176ED4) lambda(), task fired
[Log] HTMLMediaElement::mediaPlayerRenderingModeChanged(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::updateMediaPlayer(9D9CBDD4B0176ED4) 
[Info] HTMLMediaElement::webkitCurrentPlaybackTargetIsWireless(9D9CBDD4B0176ED4) false
[Info] MediaElementSession::wirelessVideoPlaybackDisabled(9D9CBDD4B0176ED4) returning false because media engine says so
[Debug] HTMLMediaElement::dispatchEvent(9D9CBDD4B0176ED4) stalled

iOS logs when loading the page

[Log] MediaSessionManageriOS::sessionCanProduceAudioChanged(0) 
[Log] MediaSessionManageriOS::sessionCanProduceAudioChanged(0) 
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(0), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(0), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Log] MediaSessionManageriOS::clientCharacteristicsChanged(0) (3A149048EECF0B1)

On iOS, it seems that things are working as far as DASH is concerned, as we get as far as complaining about the media codec. If I try the other streams I get mostly the same story - the browser is complaining about the audio (mp4a.40.2). However, that format should actually be supported on iOS per https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/FrequentlyAskedQuestions/FrequentlyAskedQuestions.html.

iOS logs when playing tears of steal

[Warning] compat: element VIDEO does not support any of these events: fullscreenchange, webkitfullscreenchange, mozfullscreenchange, msfullscreenchange, MSfullscreenchange, FullscreenChange, webkitFullscreenChange, mozFullscreenChange, msFullscreenChange, MSFullscreenChange
[Info] API: Calling loadvideo – "https://www.bok.net/dash/tears_of_steel/cleartext/stream.mpd" – "dash"
[Info] DRM: Clearing-up DRM session.
[Info] DRM: Nothing to clear. Returning right away. No state = – true
[Info] API: playerStateChange event – "LOADING"
[Info] Init: Clearing HTMLMediaElement's src
[Info] Init: Creating MediaSource
[Info] Init: Attaching MediaSource URL to the media element – "blob:https://stk8v3sx-8000.euw.devtunnels.ms/8df9321c-76c3-4217-811c-36e836192000"
[Log] MediaSessionManageriOS::addSession(0) (CEB36967233B5CE9)
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(1), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(1), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Warning] DASH: No usable id found in the Period. Generating one.
[Info] Unsupported Representation – "audio/mp4;codecs=\"mp4a.40.2\"" – "audio/en" – 132348
[Info] Init: Clearing HTMLMediaElement's src
[Info] DRM: Clearing-up DRM session.
[Info] DRM: Nothing to clear. Returning right away. No state = – true
[Error] API: The player stopped because of an error – MediaError: MediaError (MANIFEST_PARSE_ERROR) No supported audio adaptations
MediaError: MediaError (MANIFEST_PARSE_ERROR) No supported audio adaptations
	(anonym funktion) (public_api.ts:477)
	(anonym funktion) (event_emitter.ts:99)
	forEach
	trigger (event_emitter.ts:97)
	(anonym funktion) (media_source_content_initializer.ts:112)
	(anonym funktion) (event_emitter.ts:99)
	forEach
	trigger (event_emitter.ts:97)
	_onFatalError (manifest_fetcher.ts:545)
[Info] API: playerStateChange event – "STOPPED"
[Error] Unhandled Promise Rejection: CancellationError: This task was cancelled.
	onCancellablePromiseCancellation (create_cancellable_promise.ts:53)
	(anonym funktion) (task_canceller.ts:216:82)
	trigger (task_canceller.ts:302)
	cancel (task_canceller.ts:182)
	dispose (media_source_content_initializer.ts:151)
	(anonym funktion) (public_api.ts:554)
	(anonym funktion) (task_canceller.ts:216:82)
	trigger (task_canceller.ts:302)
	cancel (task_canceller.ts:182)
	(anonym funktion) (public_api.ts:474)
	(anonym funktion) (event_emitter.ts:99)
	forEach
	trigger (event_emitter.ts:97)
	(anonym funktion) (media_source_content_initializer.ts:112)
	(anonym funktion) (event_emitter.ts:99)
	forEach
	trigger (event_emitter.ts:97)
	_onFatalError (manifest_fetcher.ts:545)

If I try to play the Google - Sintel webm only on iOS, which should also be supported, I get the following logs.

iOS logs when playing Sintel (webm)

[Warning] compat: element VIDEO does not support any of these events: fullscreenchange, webkitfullscreenchange, mozfullscreenchange, msfullscreenchange, MSfullscreenchange, FullscreenChange, webkitFullscreenChange, mozFullscreenChange, msFullscreenChange, MSFullscreenChange
[Info] API: Calling loadvideo – "https://storage.googleapis.com/shaka-demo-assets/sintel-webm-only/dash.mpd" – "dash"
[Info] DRM: Clearing-up DRM session.
[Info] DRM: Nothing to clear. Returning right away. No state = – true
[Info] API: playerStateChange event – "LOADING"
[Info] Init: Clearing HTMLMediaElement's src
[Info] Init: Creating MediaSource
[Info] Init: Attaching MediaSource URL to the media element – "blob:https://stk8v3sx-8000.euw.devtunnels.ms/e9d7d2e9-0ca0-4fcd-9b99-cb59b6f4e27e"
[Log] MediaSessionManageriOS::addSession(0) (7DE295DADDBD6B1A)
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(1), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Log] MediaSessionManageriOS::updateSessionState(0) types: AudioCapture(0), AudioTrack(0), Video(1), Audio(0), VideoAudio(0), WebAudio(0)
[Log] MediaSessionManageriOS::updateSessionState(0) setting category = None, mode = Default, policy = Default, previous category = None
[Info] Parser utils: found Period with no segment.  – "Going to next one to calculate first position"
[Info] Parser utils: found Period with no segment.  – "Going to previous one to calculate last position"
[Info] MF: Manifest parsed in 19.000000000003638ms

What stands out to me here is that everything seems to be going fine, but nothing really happens after the manifest has been parsed.

Resources:

Overview by bitmovin: https://bitmovin.com/managed-media-source/
Changelog: https://developer.apple.com/documentation/ios-ipados-release-notes/ios-ipados-17-release-notes#New-Features

Answer 1 · 2023-09-28T15:22:46.000Z

Hello,

Ah yes I've heard of it but didn't perform tests with it, thanks!
If backward compatible, we could just perform the modifications you've done as a first solution, then maybe having some kind of abstraction layer in the future to better handle ManagedMediaSource's specific events.

For the tears of steal example, that's weird, it's true that maybe mp4 containers are not supported? It would still be weird because now HLS streams allow some mp4-based containers, so I suppose IOS should already have support for those. Maybe it is the way it is signaled (audio/mp4;codecs="mp4a.40.2", through here ManagedMediaSource.isTypeSupported) they don't like?

You could try calling:

ManagedMediaSource.isTypeSupported(`audio/mp4;codecs="mp4a.40.2"`);

in a console or standalone page and see if it returns true.

For the webm stream that's weird. I would guess the "sourceopen" (https://w3c.github.io/media-source/#dfn-sourceopen) event is not sent here? I don't know what would trigger it through a ManagedMediaSource and why it would be different than for a regular MediaSource... Maybe you're not supposed to attach a MediaSource to an HTMLMediaElement through an object URL created through the URL.createObjectURL API?

I'll have to look at Apple doc and examples if they exist.

Answer 2 · 2023-10-03T15:10:42.000Z

macOS reports true and iOS false for audio/mp4;codecs="mp4a.40.2". So perhaps ManagedMediaSource is not to blame there.

For webm, which should work, you're right about sourceopen not being called.

Answer 3 · 2023-10-09T14:48:22.000Z

OK so we have to look at why it doesn't trigger there.

Strangely, Webkit's own tests seem to not expect anything else (https://github.com/WebKit/WebKit/blob/8d7fefde87632cff6a30871a6bd46a7c37cf1470/LayoutTests/media/media-source/media-managedmse.html#L34).
Maybe we could test a subset of that page to see if it works.

Answer 4 · 2023-10-24T15:30:29.000Z

Hello @alexg-axis,

We have made some tests on Safari 17 using ManagedMediaSource instead of MediaSource and it seems to work fine.

In this article Apple explain some details:

When designing Managed MSE, we wanted to make sure that nothing was left out by accident and that users continue to get the same level of features as they did in the past. So to activate Managed MSE on Mac, iPad, and iPhone, your player must provide an AirPlay source alternative. You can still have access to Managed MSE without it, but you must explicitly disable AirPlay by calling disableRemotePlayback on your media element from the Remote Playback API

Have you disabled the remote playback ? Can you try to disable it and give us feedback ?

video.disableRemotePlayback = true;

Answer 5 · 2023-10-25T06:31:30.000Z

Thank you for your help. I'm happy to report that it works with the WebM stream now! For others reading this it might be worth clarifying that currently you need to enable the Managed Media Source Extension feature flag on iOS in Safari's advanced settings. There's a flag for requiring an AirPlay source which is enabled by default, which sounds like why we need to disable remote playback.

What would be the way forward for getting this in rx-player? As it currently requires the feature flag and disabling remote playback, it's rather obtrusive. But as DASH currently doesn't work at all on iOS, perhaps it is safe to simply include ManagedMediaSource at the bottom of the list of media source implementation and then documenting the behavior?

Again, thank you for your help. I'm excited by the near future of DASH on iOS.

Answer 6 · 2023-11-01T11:48:22.000Z

Apple has released iOS 17.1 and with it comes support for all the codecs one would expect. That is, WebM isn't the only working codec any more! I'm unsure if the feature flag is still required as I already had it on before upgrading.

Answer 7 · 2023-11-03T18:12:48.000Z

Apple has released iOS 17.1 and with it comes support for all the codecs one would expect

Oh nice, that was a big pain point we had when testing it and thinking of potential use cases.

What would be the way forward for getting this in rx-player?

We also thought about this and for now, as it is only under a feature flag (for the main use case for it) and as the specification for the ManagedMediaSource API is still under discussion (w3c/media-source#320, w3c/media-source#329) for its inclusion in a future MSE recommendation, we were waiting until this API is well-specified and more signal about its stability is given before including it in the RxPlayer.

We do think it will be interesting to include it and perhaps rely on its other features when it becomes an official and stable part of MSE though.

We also wondered whether it was our role to manually set the disableRemotePlayback property to true or to let the application do it. Here also, the EME recommendation is still a "recommendation draft" anyway (https://www.w3.org/standards/history/remote-playback/) so we moved that subject for later.

Answer 8 · 2023-11-03T21:15:55.000Z

macOS reports true and iOS false for audio/mp4;codecs="mp4a.40.2". So perhaps ManagedMediaSource is not to blame there.

For webm, which should work, you're right about sourceopen not being called.

the inability to use mp4 in ManagedMediaSource should be fixed in iOS 17.1

Answer 9 · 2023-11-03T21:19:38.000Z

OK so we have to look at why it doesn't trigger there.

ManagedMediaSource in Safari requires that you either make it compatible with AirPlay (by adding a secondary source element with an AirPlay compatible stream: such as plain mp4, hls.

Or to explicitly disable remote playback by adding the disableRemotePlayback attribute to your video element (e.g. video.disableRemotePlayback = true;

See source code of https://jyavenard.github.io/htmltests/tests/ManagedMediaSource/bipbop.html

Answer 10 · 2023-11-03T21:20:24.000Z

Apple has released iOS 17.1 and with it comes support for all the codecs one would expect. That is, WebM isn't the only working codec any more! I'm unsure if the feature flag is still required as I already had it on before upgrading.

no, changing the feature flag is not longer required, it's on by default from iOS 17.1