[Question] MediaCodecVideoRenderer - on what basis is the late frame threshold determined

[KarlGallagher]

Hi,

We are looking into some of the ExoPlayer implementation around the MediaCodec API. I have noticed that the threshold for when a decoded frame is considered to have 'late' availability is 30ms ('earlyUs' variable)

So I have two questions:

1.)How was the 30ms threshold derived
2.)What would the likely impact be if it was increased - lets say, to 90ms.

The context to this question is that we are currently observing MediaCodec stalls when the decoder is reconfigured during a representation change (in which both bitrate and resolution and/or frame rate changes). This is resulting in dropped frames for us and on some devices visible glitches.

[botaydotcom]

Hi @karlg-arris,

@ojw28 maybe able to answer this question better, but I believe the answers are:

1. It's arbitrarily decided.
2. It may reduce the number of frame drops, but the video quality is not improved (and maybe much worse). More video frames are shown but frames will be shown at non-constant rate. Moreover, increasing this threshold will also force the MediaCodec to render more frames when it already couldn't keep up in the first place, which leads to more CPU starving, and more subsequent late frames.

Some very similar questions have been discussed on this issue: #3040 (you can search for 30ms further down the page).

[KarlGallagher]

Hi @botaydotcom

I read that thread previously. It seems the concern there was around generally slow codecs or those that used significant system resources.

The example I am providing is based around a very specific stall immediately subsequent to a decoder reconfiguration on devices that otherwise do not exhibit decoder performance issues - e.g. dropped frames even at low bitrates (400kbps).

Given that use case might an increase in the threshold help more than hinder?

[ojw28]

Given that use case might an increase in the threshold help more than hinder?

Am I right in saying that we're talking about comparing (a) rendering some frames late (possibly in rapid succession as we catch up to the audio), vs (b) dropping the late frames (which likely allows the catch up to occur more quickly). Presumably "help" in this case means "looks nicer to the user". So I think to answer that question, you'd probably need to give it a try and see what it looks like.

[KarlGallagher]

@ojw28 Yes your assertion is correct. I get your point, so this really depends on observable impact rather than simply reducing the amount of log traces about dropped frames.

On a final note - is it worth raising a bug against the apparent underlying cause (decoder stalls post reconfiguration) here? or is that something that is expected / well understood already?

Thanks,
Karl.

[ojw28]

It really depends on how big the pause is, why the pause occurs, and (pragmatically) whether it's a device that the OEM is still delivering significant software updates to. The issue would need to be reported to the device manufacturer, but if you provide details here, including a full bugreport captured with "adb bugreport", we could take a quick look for anything unusual.

[KarlGallagher]

Thank you.

My questions have been answered. Thanks for taking the time out on this one.