Subtitle (Track - VTT)
talaysa opened this issue · 2 comments
Hi,
Is there any subtitle support on chromecast? We have external track with VTT format, how we can send them with chromecast?
Thanks,
Salih
Support for playing subtitles on Chromecast is a feature that I would like, as well. I spent some time yesterday exploring this issue, and will share my findings.
Ideally, I'd like this to trigger some discussion because there are a few moving parts and proper support for this feature would require touching both clappr-core
and hlsjs-playback
.
For my next step, I believe that I can write a small plugin to extend (ie: monkey-patch) this clappr-chromecast-plugin to add support for settings.playback.externalTracks
. Even if successful, it still won't be able to support in-stream text tracks.
before digging into code, lets start with some Clappr settings to test text tracks:
// -----------------------------------------------
// references:
// https://github.com/clappr/clappr/issues/1477
// -----------------------------------------------
var initialize_text_tracks = function() {
var video = document.querySelector('video')
if (!video) return
var textTracks = video.textTracks
if (!textTracks || !textTracks.length) return
for (var i=0; i < textTracks.length; i++) {
if (textTracks[i].mode === 'showing') return
}
// turn on the first subtitles text track (which is always "Disabled") to display the "CC" icon/menu in the media-control panel
textTracks[0].mode = 'showing'
}
// -----------------------------------------------
// references:
// http://demo.theoplayer.com/closed-captions-subtitles
// https://github.com/videojs/video.js/tree/v7.10.2/docs/examples/elephantsdream
// -----------------------------------------------
var player = new Clappr.Player({
source: 'https://cdn.theoplayer.com/video/elephants-dream/playlist-single-audio.m3u8', // in-stream WebVTT: Chinese, French
poster: 'https://demo.theoplayer.com/hubfs/Demo_zone/elephants-dream.jpg',
height: 360,
width: 640,
plugins: [ChromecastPlugin],
chromecast: {
appId: '9DFB77C0',
media: {
type: ChromecastPlugin.Movie,
title: 'Elephants Dream',
subtitle: '2006 Dutch computer animated science fiction fantasy experimental short film produced by Blender Foundation'
}
},
playback: {
crossOrigin: 'anonymous',
externalTracks: [{
lang: 'en-US',
label: 'English',
kind: 'subtitles',
src: 'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.en.vtt'
},{
lang: 'sv',
label: 'Swedish',
kind: 'subtitles',
src: 'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.sv.vtt'
},{
lang: 'ru',
label: 'Russian',
kind: 'subtitles',
src: 'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.ru.vtt'
},{
lang: 'ja',
label: 'Japanese',
kind: 'subtitles',
src: 'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.ja.vtt'
},{
lang: 'ar',
label: 'Arabic',
kind: 'subtitles',
src: 'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.ar.vtt'
}]
},
events: {
onPlay: initialize_text_tracks
}
});
observations without any changes to Clappr:
- on desktop Chrome browser:
- all text tracks load in Clappr and function correctly
- on Chromecast (gen 1)
- no text tracks load
- as expected, because this feature isn't implemented
- the HLS video doesn't play
- interesting, because I've never seen Chromecast refuse an HLS
- this is obviously an issue with Chromecast and its support for HLS with in-stream text tracks
- no text tracks load
necessary changes to settings to continue with testing:
var player = new Clappr.Player({
source: 'https://d2zihajmogu5jn.cloudfront.net/elephantsdream/ed_hd.mp4',
...
});
- the "Add Advanced Features" section in the Chromecast docs for building a Chrome sender app outlines how to manage (external) text tracks
- as a first step, I confirmed that the above methodology will work with the Clappr Chromecast receiver app by making the following changes to chromecast.js and testing them while running the webpack dev server:
get activeTrackIds() { let trackId = this.container.closedCaptionsTrackId return (trackId >= 0) ? [trackId] : [] } loadMedia() { this.container.pause() let src = this.container.options.src Log.debug(this.name, 'loading... ' + src) let mediaInfo = this.createMediaInfo(src) let request = new chrome.cast.media.LoadRequest(mediaInfo) request.autoplay = true request.activeTrackIds = this.activeTrackIds if (this.currentTime) { request.currentTime = this.currentTime } this.session.loadMedia(request, (mediaSession) => this.loadMediaSuccess('loadMedia', mediaSession), (e) => this.loadMediaError(e)) } createMediaInfo(src) { let mimeType = ChromecastPlugin.mimeTypeFor(src) let mediaInfo = new chrome.cast.media.MediaInfo(src) mediaInfo.contentType = this.options.contentType || mimeType mediaInfo.customData = this.options.customData let metadata = this.createMediaMetadata() mediaInfo.metadata = metadata let tracks = this.createMediaTracks() mediaInfo.tracks = tracks return mediaInfo } // --------------------------------------------- // references: // https://developers.google.com/cast/docs/chrome_sender/advanced // --------------------------------------------- createMediaTracks() { const tracks = [] const langs = [['en','English'],['sv','Swedish'],['ru','Russian'],['ja','Japanese'],['ar','Arabic']] for (let i=0; i < langs.length; i++) { const [language, name] = langs[i] const url = `https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.${language}.vtt` const track = new chrome.cast.media.Track(i, chrome.cast.media.TrackType.TEXT) track.trackContentId = url track.trackContentType = 'text/vtt' track.subtype = chrome.cast.media.TextTrackType.SUBTITLES track.language = language track.name = name track.customData = null tracks.push(track) } return tracks }
- having confirmed this will work, the next step was to reimplement
createMediaTracks
:createMediaTracks() { const textTracks = this.container.closedCaptionsTracks // [{id, name, track}] return textTracks.map(textTrack => { const track = new chrome.cast.media.Track(textTrack.id, chrome.cast.media.TrackType.TEXT) //track.trackContentId = undefined //track.trackContentType = undefined track.subtype = chrome.cast.media.TextTrackType.SUBTITLES track.language = textTrack.track.language track.name = textTrack.name track.customData = null return track }) }
- this quickly hit a roadblock
textTracks
is obtained from the getter:closedCaptionsTracks
textTracks[0].track === document.querySelector('video').textTracks[0]
- API:
- TextTrack API doesn't provide any way to obtain the URL from which it was loaded
- an in-stream HLS TextTrack doesn't include the URL to its m3u8 manifest
- an external TextTrack doesn't include the URL from its underlying
<track>
element
-
thoughts..
- external text tracks are low-hanging fruit
- their URLs can be obtained from both
settings.playback.externalTracks
and DOM<track>
elements
- their URLs can be obtained from both
- in-stream HLS text tracks are more complicated
- external text tracks are low-hanging fruit
-
external text tracks:
- problems:
- mapping from
this.container.closedCaptionsTrackId
to the correct URLthis.container.closedCaptionsTrackId
refers to the complete list ofthis.container.closedCaptionsTracks
- if there are in-stream text tracks, then they will be included and make the mapping more complicated
- TextTrack includes an
id
attribute that can be used to query the underlying<track>
element, if any- when
clappr-core
creates DOM<track>
element, it doesn't add any (unique)id
attribute
- when
- mapping from
- observations:
- by testing the aforementioned Clappr settings
settings.playback.externalTracks
are always loaded BEFORE in-stream text tracks- if this can be guaranteed, then things become much easier
- by testing the aforementioned Clappr settings
- problems:
-
in-stream HLS text tracks:
- problems:
hlsjs-playback
depends uponhls.js
- the
hls.js
API fires several interesting events- SUBTITLE_TRACK_LOADING
- provides the text track manifest's URL
- hlsjs-playback doesn't add any listener
- SUBTITLE_TRACK_LOADED
hlsjs-playback
adds a listener, but doesn't inspect the data for its underlying URLs
- SUBTITLE_TRACK_LOADING
- the
- problems:
I think that's a fairly thorough summary of where I'm at right now.
As I said, I'm going after the low-hanging fruit.. and will report back with any progress.
imho, it works great; feedback (and PRs) welcome 😃
PS, the live demo doesn't set crossOrigin
, because that would break video playback. The result is that the text tracks won't play in the web browser, but they'll play fine on Chromecast.
PS, regarding clappr#1477, the initialize_text_tracks
workaround included in the example Clappr settings (above) is no-longer needed when using this enhanced plugin; it performs this task automagically (and more cleanly).