Manifest with different subtitle track order than existing video TextTracks breaks caption display and selection

[SivanA-Kaltura]

What version of Hls.js are you using?

1.1.5

What browser (including version) are you using?

Chrome 98.0.4758.102 (x86_64)

What OS (including version) are you using?

macOS Big Sur

Test stream

https://codepen.io/sivana-kaltura/pen/oNoqwba

Configuration

{
  debug:true
}

Additional player setup steps

No response

Checklist

• The issue observed is not already reported by searching on Github under https:/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hls-js.netlify.com/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hls-js-dev.netlify.com/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Play first video
2. Go to 1:00 and verify that english captions are showing
3. Click on change media
4. Play second video
5. Go to 2:00 and verify that english captions are not showing

Expected behaviour

English captions are showing in both videos.

What actually happened?

English captions are not showing in second video.

Some additional information after debugging:
In manifest 1, english is at index 0 and swedish is at index 1
In manifest 2, german is at index 0 and english is at index 1

1. After changing media, subtitle-track-controller sets video element.textTracks[0].mode="disabled" (english)
   and video element.textTracks[1].mode="showing" (swedish)
2. subtitle-stream-controller sets currentTrackId = 1
3. When timeline-controller receives a cue for level 1, it checks
   this.textTracks[1].mode and because its "disabled", no cues are appended

Console output

It's over the max length limit :(

Chrome media internals output

No response

[mtoczko]

Hi @SivanA-Kaltura
This will help you solve the problem

hls.js/demo/main.js

Lines 340 to 376 in 492a103

	onDemoConfigChanged(true);
	console.log('Using Hls.js config:', hlsConfig);
	self.hls = hls = new Hls(hlsConfig);
	logStatus('Loading manifest and attaching video element...');
	const expiredTracks = [].filter.call(
	video.textTracks,
	(track) => track.kind !== 'metadata'
	);
	if (expiredTracks.length) {
	const kinds = expiredTracks
	.map((track) => track.kind)
	.filter((kind, index, self) => self.indexOf(kind) === index);
	logStatus(
	`Replacing video element to remove ${kinds.join(' and ')} text tracks`
	);
	const videoWithExpiredTextTracks = video;
	video = videoWithExpiredTextTracks.cloneNode(false);
	video.removeAttribute('src');
	video.volume = videoWithExpiredTextTracks.volume;
	video.muted = videoWithExpiredTextTracks.muted;
	videoWithExpiredTextTracks.parentNode.insertBefore(
	video,
	videoWithExpiredTextTracks
	);
	videoWithExpiredTextTracks.parentNode.removeChild(
	videoWithExpiredTextTracks
	);
	}
	addChartEventListeners(hls);
	addVideoEventListeners(video);
	hls.loadSource(url);
	hls.autoLevelCapping = levelCapping;
	hls.attachMedia(video);

[SivanA-Kaltura]

Thanks @mtoczko.
This can be a workaround, however it can introduce a different set of problems, depending on the page a player in hosted on. In most cases it's also not required if the order of the subtitle tracks did not change.

Since hls.js reuses text tracks in timeline-controller, this could be resolved by mapping a video element track index to a manifest track index, or by allowing track selection by name/label and not by index, but it's a design decision whether to fix or to leave as a limitation.

[robwalch]

Tested this against v1.2.9 and found another issue. The first HLS produces an English TextTrack of kind "captions" but the second HLS expects one of kind "subtitles". This may be a regression introduced after this issue was reported, made to match the track kind values produced by Safari.

Please have a look at how the change in this branch changes the outcome: https:/video-dev/hls.js/compare/bugfix/texttrack-reuse-mixed-kind.

HLS.js will respond to TextTrack state changes so that track selection can be performed in the DOM or on the HTMLMediaElement. The state of reused tracks or a change made in setup may be overriding the default HLS subtitles for a new player instance > loadSource.

If video element reuse and native TextTrack rendering and selection is something you need, please consider contributing a fix. If not, the renderTextTracksNatively configuration options disables HLS.js's use of TextTracks completely so that applications can handle subtitles and captions themselves.

[giladna]

@robwalch do you think this issue will have a fix in upcoming versions?

[robwalch]

do you think this issue will have a fix in upcoming versions?

Would Kaltura be willing to contribute a fix?

[robwalch]

Hi @giladna and @SivanA-Kaltura,

#5371 is a start. The default media option is selected after calling hls.loadSource again on the same player instance with the same video element even when the index of that option is different than the it was in the previous stream. It looks like there were just a couple flags preventing default track selection from happening when using a single instance to load multiple sources. I think with the latest release if you used a new player instance for each asset this would not be an issue.

Sorry I spoke too soon. The issue with default track selection is just the tip of the iceberg.