Add preferred mime type(s) to DefaultTrackSelector parameters

[ybai001]

I'd like to implement a customized TrackSelector. My situations are

1. I don't want to change any code of ExoPlayer core library so that I'll not modify DefaultTrackSelector directly;
2. I know I can do some kinds of customization via setParameter API.

trackSelector.setParameters( trackSelector .buildUponParameters() .setMaxVideoSizeSd() .setPreferredAudioLanguage("deu"));

But Current APIs can't satisfy my requirement. For example, I'd like to always set higher priority for Dolby Digital plus track than AAC audio track. It seems that current interface can't support it.

According to ExoPlayer official documentation (https://exoplayer.dev/customization.html),

TrackSelector – Implementing a custom TrackSelector allows an app developer to change the way in which tracks exposed by a MediaSource are selected for consumption by each of the available Renderers.

My first feeling is I need to copy all files under "library\core\src\main\java\com\google\android\exoplayer2\trackselection" folder into my application layer and then do some modification. Do you have any suggestion for this? Is there a better solution?

Thank you very much in advance.

[tonihei]

It's always possible just to fork and replace DefaultTrackSelector and set your customized track selector in SimpleExoPlayer.Builder.setTrackSelector.

However, we usually advise against this because DefaultTrackSelector is quite complex and as soon as we start making further updates you'd need to constantly merge in our changes to keep your modifications. Instead it's much better if we could support your requirements by parameters directly.

How do your audio tracks actually differ? Do they have different mime types, samples rate, channel counts, bitrates?
If they differ reliably in any of those dimensions, it should be straight-forward to achieve your preference ordering. Actually, audio tracks are already ordered by channel count, sample rate and bitrate (in that preference order).

Another aspect that may be relevant is whether your device you are testing on supports playing the Dolby Digital plus track. If the track is not supported by the device, we'd always prefer a supported track if it exists (like AAC).

Finally, there are some less intrusive alternatives to forking the entire DefaultTrackSelector class:

• You could subclass DefaultTrackSelector and only override selectAudioTrack. In there you can do whatever evaluations and preference ratings you prefer and only return your selected track.
• Another alternative is to switch tracks manually after playback started. You can specify any override by calling DefaultTrackSelector.ParametersBuilder.setSelectionOverride and specifiying the track you want to select. However, this is only possible after the media has been prepared (listen to onTracksChanged) and will cause an audible pause if playback already starting playing.

[ybai001]

Hi, tonihei,

Thank you very much for this detailed answer. It's very helpful. Honestly, I have never anticipated that I can get so detailed answer.

The case of us is there are several audio tracks in the DASH manifest. There are encoded by AAC or Dolby AC-4 with different languages.

The requirements are
(1) If both AAC and AC-4 have some kind of languages, e.g. English, application always select AC-4 audio track whatever the channel count/bitrate/sample rate are.
(2) If only AAC has some specific language track but AC-4 doesn't, AAC audio track will be selected.

If you team can support this interface, such as

trackSelector.setParameters( trackSelector .buildUponParameters() .setPreferredCodecs("ac-4.02.02.00"));
or
trackSelector.setParameters( trackSelector .buildUponParameters() .setPreferredMimetype("audio/ac4"));

It will be good to us. The example DASH manifest is

<AdaptationSet mimeType="audio/mp4" segmentAlignment="true" startWithSAP="1"> <!-- <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/> --> <SegmentTemplate duration="4000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/seg-$Number$.m4s" startNumber="1" timescale="1000"/> <Representation audioSamplingRate="48000" bandwidth="72960" codecs="ac-4.02.02.00" id="audio/und/ac-4"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:mpegB:cicp:ChannelConfiguration" value="2"/> <SupplementalProperty schemeIdUri="tag:dolby.com,2016:dash:virtualized_content:2016" value="1"/> </Representation> </AdaptationSet> <AdaptationSet mimeType="audio/mp4" segmentAlignment="true" startWithSAP="1"> <SegmentTemplate duration="4000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/seg-$Number$.m4s" startNumber="1" timescale="1000"/> <Representation audioSamplingRate="48000" bandwidth="114421" codecs="mp4a.40.2" id="audio/und/mp4a"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> </Representation> </AdaptationSet>

In fact, I have modified compareTo(AudioTrackScore other) method in DefaultTrackSelector to make this work. But I understand this is much more difficult since you need to consider more situations from common platform perspective. Due to customer doesn't allow us to modify core library code, I can't use this method.

And I used setSelectionOverride() in application layer and it can work. But I don't like this method since there is side-effect as you said.

I'll try "override selectAudioTrack" method.

Thanks a lot again.

BR,
Yanning

[ybai001]

Hi, tonihei,

I tried your proposal 1:

You could subclass DefaultTrackSelector and only override selectAudioTrack. In there you can do whatever evaluations and preference ratings you prefer and only return your selected track.

It can work!

But there is a block point: selectAudioTrack() needs to access "private" method getAdaptiveAudioTracks(). So that I have to change one line code in defaultTrackSelector. Is it possible for your team to change this method from "private" to "protected"?

BR,
Yanning

[tonihei]

The requirements are
(1) If both AAC and AC-4 have some kind of languages, e.g. English, application always select AC-4 audio track whatever the channel count/bitrate/sample rate are.
(2) If only AAC has some specific language track but AC-4 doesn't, AAC audio track will be selected.

Just to ensure I understand this requirement correctly: Your first criterium is the language. If the track's language matches your preference, it should always be preferred over other tracks no matter the codec or mime type. I think this part of the requirement can already be fulfilled by explicitly setting your preferred language (e.g. setPreferredAudioLanguage("en")). If two tracks have your preferred language (or both haven't), you'd like to prefer AC-4 mime types over AAC. Is this summary correct?

Side note: Your example manifest snippet doesn't contain language tags. Not sure if this was supposed to show the language selection requirement somehow.

setPreferredMimeType("audio/ac4")

I think we should just add this option to accommodate your requirement. For the preference ordering it should probably come after preferred languages and after specific format constraints (like maxAudioBitrate, maxAudioChannelCount), but before all the implicit ordering (like system Locale language or ordering by channel count, sample rate and bitrate). Would that work for you?

But there is a block point: selectAudioTrack() needs to access "private" method getAdaptiveAudioTracks(). So that I have to change one line code in defaultTrackSelector. Is it possible for your team to change this method from "private" to "protected"?

I think the idea would be that you just return a single Format for your preferred track. If you want to support audio adaptation with your special constraints you'd need to filter the formats yourself I think because there is no guarantee that getAdaptiveAudioTracks is compatible with whatever logic you implement in selectAudioTrack. Also note that adaptive audio adaptation is currently only turned on by default if there is no video and audio adaptation can be seamless. So it may actually never be relevant for your case?

[ybai001]

Just to ensure I understand this requirement correctly: Your first criterium is the language. If the track's language matches your preference, it should always be preferred over other tracks no matter the codec or mime type. I think this part of the requirement can already be fulfilled by explicitly setting your preferred language (e.g. setPreferredAudioLanguage("en")). If two tracks have your preferred language (or both haven't), you'd like to prefer AC-4 mime types over AAC. Is this summary correct?

Your understanding is fully correct.

Side note: Your example manifest snippet doesn't contain language tags. Not sure if this was supposed to show the language selection requirement somehow.

Sorry for confusion. This request is from customer and I have no that kind of manifest in my hand. Just provided a "simple" version without "language" tags.

I think we should just add this option to accommodate your requirement. For the preference ordering it should probably come after preferred languages and after specific format constraints (like maxAudioBitrate, maxAudioChannelCount), but before all the implicit ordering (like system Locale language or ordering by channel count, sample rate and bitrate). Would that work for you?

It works for us. And

1. I think the API could be
   setPreferredMimeType(String mimeType)
2. Just want to double confirm: it is not format.containerMimeType (in my example manifest, it is "audio/mp4") and should be format.sampleMimeType (in my example case, it is "audio/ac4")
3. I'm not sure which proposal is better: setPreferredMimeType(String mimeType) or setPreferredMimeTypes(String[] mimeTypes) For example, prefering "eac3", "eac3-joc", "ac-4" over "aac" while "eac3", "eac3-joc" and "ac-4" have same "MIME type based" priority (so that channel count/sample rate/... will decide the final priority of eac3/eac3-joc/ac-4) . That is to say, the parameter is a string[]. Proposal 1 is easier while proposal 2 is more general. Both proposals can satisfy our current customer requirement so that you may decide it from platform perspective. :)

I think the idea would be that you just return a single Format for your preferred track. If you want to support audio adaptation with your special constraints you'd need to filter the formats yourself I think because there is no guarantee that getAdaptiveAudioTracks is compatible with whatever logic you implement in selectAudioTrack. Also note that adaptive audio adaptation is currently only turned on by default if there is no video and audio adaptation can be seamless. So it may actually never be relevant for your case?

I think you are right.

Again, thanks a lot for your great help. I truly appreciate it very much. Your team is always warm-hearted and very efficient. Hope you and your family are healthy in this special period.

[tonihei]

I'm not sure which proposal is better: setPreferredMimeType(String mimeType) or setPreferredMimeTypes(String[] mimeTypes) For example, prefering "eac3", "eac3-joc", "ac-4" over "aac" while "eac3", "eac3-joc" and "ac-4" have same "MIME type based" priority (so that channel count/sample rate/... will decide the final priority of eac3/eac3-joc/ac-4) . That is to say, the parameter is a string[]. Proposal 1 is easier while proposal 2 is more general. Both proposals can satisfy our current customer requirement so that you may decide it from platform perspective

In a similar case for languages we have two methods setPreferredAudioLanguage(String) and setPreferredAudioLanguages(String...), which both end up in an ImmutableList<String> in the Parameters object (with one or many elements). I think we can just do the same here. I'll mark as an enhancement for now. Are you thinking about providing a PR yourself or shall we implement this?

[ybai001]

I hope your team can help to implement it since it will be more quickly to be merged. Thanks a lot in advance.