MediaSessionActionHandler doesn't work for seek operations

[ChunMinChang]

When I tried implementing this spec in Firefox, I had trouble to meet the following requirement:

Run handler with the details parameter set to:

• MediaSessionSeekActionDetails, if action is seekbackward or seekforward.
• MediaSessionSeekToActionDetails, if action is seekto.
• Otherwise, with MediaSessionActionDetails.

Specifically, the following javascript doesn't work

navigator.mediaSession.setActionHandler("seekto", function(details) {
  console.log(details.action); // ok, it's "seekto".
  console.log(details.seekTime); // undefined, but it's a `required` members in MediaSessionSeekToActionDetails
});

The problem is because the details is defined as MediaSessionActionDetails instead of MediaSessionSeekToActionDetails.

callback MediaSessionActionHandler = void(MediaSessionActionDetails details);

There's no virtual function table in dictionaries, so the javascript has no idea the details is actually a MediaSessionSeekToActionDetails.

See more details here.

[ChunMinChang]

There are three possible workarounds I could think for now:

1. Define MediaSessionActionHandler to void(Object details)
2. Use a single MediaSessionActionDetails containing optional members for seek operations
3. Use specific setActionHandlers for seek operations

1. Define MediaSessionActionHandler to void(Object details)

Accessing Object directly is likely to cause security issues because JS objects are highly configurable. However, the details is never used as an input argument. It's created in the browser itself so it might be ok to do so.

2. Use a single MediaSessionActionDetails containing optional members for seek operations

By merging MediaSessionSeekActionDetails and MediaSessionSeekToActionDetails into MediaSessionActionDetails, details.seekTime becomes a valid member for MediaSessionActionHandler

dictionary MediaSessionActionDetails {
  required MediaSessionAction action;
  double seekOffset; // optional
  double seekTime; // optional
  bool fastSeek; // optional
};

However, it requires some additional statements in the spec like:

• seekTime MUST be set when it's for seekto's action handler
• seekOffset and seekTime MAY be set when it's for the action handlers of seekbackward or seekforward
• ...

3. Use specific setActionHandlers for seek operations

For example,the following are specific setActionHandler and ActionHandler for seekbackward, seekforward,

callback MediaSessionSeekActionHandler = void(MediaSessionSeekActionDetails details);

void setSeekActionHandler(MediaSessionAction action, MediaSessionSeekActionHandler? handler);

By defining above interface, the javascript side can know the type of the details exactly. However, it's verbose and there might be lots of customized setActionHandler and ActionHandler with MediaSessionAction grows.

[adamopenweb]

@mounirlamouri or @beccahughes, is it possible to have someone look at this issue? As we're working on MediaSession in Firefox and need to move forward soon. (Not sure the best way to bring this up, but appreciate any help.)

[beccahughes]

Can the javascript not know the details is a MediaSessionSeekToActionDetails by checking action == seekto?

[ChunMinChang]

Can the javascript not know the details is a MediaSessionSeekToActionDetails by checking action == seekto?

The details defined in the MediaSessionActionHandler is a MediaSessionActionDetails, rather than MediaSessionSeekToActionDetails. It won't be changed no matter what action defined in MediaSessionAction is used in setActionHandler.

[bzbarsky]

@beccahughes If the IDL says MediaSessionActionDetails then the algorithm at https://heycam.github.io/webidl/#invoke-a-callback-function will treat the dictionary it's given as a MediaSessionActionDetails, and will produce a JS object with just a single property named "action". So the JS can tell that the action is "seekto", but it can't get the "seekTime" property, for example.

In particular, the spec's "Run handler with the details parameter set to" bit is not actually implementable with a Web IDL callback. What you can do with a callback is https://heycam.github.io/webidl/#invoke-a-callback-function and that will use the type the callback is declared with.

[smaug----]

@beccahughes @mounirlamouri
would be nice to get some resolution for this.

I'm trying to review a Firefox patch for this, but it is hard, when one doesn't know in which way the spec will be fixed.

[beccahughes]

My preference would be (2) since that is closest to what we have now

[jernoble]

Bikeshedding, I'd prefer something like:

dictionary MediaSessionActionDetails {
  required MediaSessionAction action;
  MediaSessionSeekActionDetails seekto; // optional
};

As opposed to hanging all the optional parameters directly off the MediaSessionActionDetails dictionary. I'm worried that more actions are going to need their own parameters, and this gives us a clean way of adding entire parameter namespaces to the MediaSessionActionDetails dictionary. The call site would look like:

navigator.mediaSession.setActionHandler("seekto", function(details) {
  if (details.action == 'seekto') {
    performSeekOperation(details.seekto.seekTime);
  }
});

[beccahughes]

Since we have the shipped in Chrome I would avoid changing this in a breaking way

https://web.dev/media-session/

[jernoble]

Since we have the shipped in Chrome I would avoid changing this in a breaking way

That's the danger of shipping enabled-by-default features that are not fully specified and don't have multiple implementations.

That said, I'm sure Chrome could come find a way to avoid breaking existing clients and also adopt whatever changes this standards body comes up with.

[ChunMinChang]

Firefox is going to use the (2) for now and soon will turn on the media-session in Firefox Nightly.

Personally I'd like to have something like what jernoble proposed in #234 (comment), but Firefox will keep the same javascript interfaces for now.

[jernoble]

Ok. It might be nice to file a bug on WebIDL detailing the issues you had with implementation, specifically around "virtual dictionaries"; this kind of thing will crop up again in the future.

[ChunMinChang]

Thanks for fixing this issue!

However, I am not sure I follow the new interface:

dictionary MediaSessionActionDetails {
  required MediaSessionAction action;
  double? seekOffset;
  double? seekTime;
  boolean? fastSeek;
};

I don't understand why seekOffset, seekTime, and fastSeek all nullable and optional at the same time. For example, the seekTime now must be provided from the user agent(browser), but what does a null seekTime mean? If a null seekTime value is used to hint the call site to ignore this value, then returning a MediaSessionActionDetails without seekTime seems more straightforward.

Is there a case demonstrating when a null seekTime or seekOffset gives a better usage over than a MediaSessionActionDetails without seekTime or seekOffset?

(or do I need to open another issue?)