Allow cue end time to be updated

[rjksmith]

Use Case
A user wants to display content synchronised to a web media object which is streaming live, and adds a suitable cue. After the cue has been added, a change occurs in real time which affects the content displayed and could not have been known beforehand.

A number of possible changes have been identified (thanks to @mavgit):

• cue end time goes from undefined to defined
  • for example, a sports game, after the undefined game time elapses, goes from undefined to a defined remaining time for post game interviews.
• cue end time goes from defined to undefined
  • for example, an award show running overtime
• cue end time going from one defined time to another defined time
  • for example, a regularly scheduled one hour newscast extended by 30 minutes due to breaking news or one hour program cut short by unscheduled government speech.

Proposal
It is proposed that the API should allow TextTrackCue.endTime to be changed after a cue has been added.

Related Issues

• Add recommendation for events with unknown end time

[rjksmith]

I believe that the existing API can support this as follows:

a. The TextTrackCueList is an existing interface, so cues can be read from the TextTrack object.

b. The TextTrackCue object exposes the endTime attribute which can be set.

c. Hence, cues can be updated by removing, updating and then adding them back to the list. For example:

function updateCueEnd(textTrack, cueIndex, endTime) {
  // read cue to update
  var cue = textTrack.cues[cueIndex];

  // remove cue
  textTrack.removeCue(cue);

  // update end time
  cue.endTime = endTime;

  // add updated cue
  textTrack.addCue(cue);
}

Best practices for metadata text tracks describes a recommendation for handling scoring for a sports event and considers the live streaming case, which offers another good use case for updating media timed events. At the start of the match the score is '0-0', which could change to either '1-0' or '0-1' at an unknown time in the future.

[nigelmegitt]

@rjksmith If removing a cue from the text track after its begin time means that its onexit() handler is run this wouldn't work for the use case, which needs to allow the end time of an active cue to be modified. I'm not sure if this is the case with your proposal or not, just saying it would be worth checking.

[chrisn]

I have done some experiments with this, will post the results here soon.

[mavgit]

Actually, once we have #8, where TextTrackCue.endTime value of Infinity represents the end of media time, all four of my common use cases above can be solved without either resetting TextTrackCue.endTime or replacing the TextTrackCue.

We just need to encourage a best practice that any TextTrackCue that is meant to run to the end of a program be set to Infinity instead of any finite time value. Then the TextTrackCue.endTime will automatically end at the same time as the 'media.duration'.

• a sports game, after the undefined game time elapses, goes from undefined to a defined remaining time for post game interviews (TextTrackCue.endTime = Infinity doesn't need to change).
• an award show running overtime (TextTrackCue.endTime = Infinity doesn't need to change)
• a regularly scheduled one hour newscast extended by 30 minutes due to breaking news (TextTrackCue.endTime = Infinity doesn't need to change)
• a one hour program cut short by unscheduled government speech. (TextTrackCue.endTime = Infinity doesn't need to change)

[chrisn]

@mavgit A slight variation on those scenarios you describe, but if those programs are followed by some other program, then we would need to update the cue end time, to ensure the cue doesn't overlap the following program.

[mavgit]

@chrisn The resource selection algorithm destroys all text track cues when the HTMLMediaElement is destroyed. So, a cue cannot outlast its containing media.duration.

[rjksmith]

@nigelmegitt Firstly, there is no stated requirement that the cue is active, although that case should certainly be included. For an active cue, I would expect that onexit() would be called as a result of removeCue and then onenter() would be called as a result of addCue in the example code so the net effect is that the cue would still be active, which meets the requirements stated in the use cases above.

If there are additional use cases which you think should be covered, please add them to this issue for discussion.

[nigelmegitt]

@rjksmith calling onexit() followed by onenter() would not meet the use case at the top of this issue, because in a live scenario that would almost certainly remove the cue's presentation and add it again. There could be a flicker or change in layout of the visible subtitle, which would be an accessibility problem - especially if the text is going to a screen reader, which would probably read it again every time this happens. Essentially the observed behaviour would not be one cue whose duration is extended, but of multiple cues containing the same content.

@mavgit a single player in a <video> element may be used to present multiple contiguous programmes, without being destroyed in between, so I think @chrisn 's #9 (comment) does need further consideration.

[rjksmith]

@mavgit @chrisn I agree with your comments. I think it depends on whether the programmes are treated as separate files/streams or whether the channel broadcasts a single never-ending stream to which files/streams are appended. Both are possible, but perhaps that should be an issue addressed in best practice.

[rjksmith]

@nigelmegitt Thanks for adding these new use cases. I acknowledge the issue you've highlighted, but I disagree with your conclusion. There are a number of ways in which the requirements can be interpreted and we should be careful to guard against adding unnecessary requirements which do not address the cause of the problem.

Flicker, layout change and re-reading text are implementation issues, and could also occur as a result of a cue end time update. For example, a request is made (asynchronously) to update an active cue's end time just as it is about to expire. Before the update can be executed, the cue end time passes and it becomes inactive, calling onexit(). The cue is then updated with a later end time and becomes active again so onenter() is called, and we have exactly the same issue, i.e. multiple active cues with the same content. If this is an identified problem, as in your accessibility use cases, the application code should be properly designed to handle these circumstances.

[rjksmith]

Following further discussion of the events issue in the Media Timed Events telecon on 16 March 2020, I've raised it as a separate item as promised.