ExoPlayer sample app only renders a few frames and then video rendering freezes

[rtamin]

Issue description

I have a video sample I am trying to play with the ExoPlayer sample app. The sample has video codec is H264 and audio codec is AC3. I am fully aware that audio will not play because ExoPlayer does not support AC3 codec. I was able to play the audio after enabling the ffmpeg extensions in ExoPlayer. The problem I am having is with rendering the video.

Initially, the video renderer would not render at all. When I try to play the sample, I would just get a black screen. In order to get video rendering to start I had to set FLAG_ALLOW_NON_IDR_KEYFRAMES when instantiating DefaultTsPayloadReaderFactory.java.

After doing that, the video will only render a few frames and then the video renderer will just freeze and stop rendering the video.

I am trying to figure out what I need to do to get the video to play.

Reproduction steps

I have provided a link where the sample video I am testing with can be downloaded. To reproduce in ExoPlayer sample app:

• Modify DefaultTsPayloadReaderFactory.java to have flag FLAG_ALLOW_NON_IDR_KEYFRAMES to always be set.

• Play the sample video in ExoPlayer sample app.

Link to test content

The video can be downloaded from my google drive:

https://drive.google.com/file/d/0B1yolYE4ZjYGVVB2cHUtZnFuVGc/view?usp=sharing

Version of ExoPlayer being used

I am using version 2.4.1.

Device(s) and version(s) of Android being used

I am testing on a device that has Android 5.0.2 and API 21.

A full bug report captured from the device

Here is a caption of "adb logcat" after I try to play the video in ExoPlayer sample app and then I eventually stop the application. I have attached the output of "adb bugreport" with this ticket as an attachment.
bugreport.txt
bugreport.txt

I/ExoPlayerImpl: Init ExoPlayerLib/2.4.1 [cnt, CNT, boundary, 21]
D/EventLogger: state [0.01, true, I]
D/EventLogger: state [0.04, true, B]
D/EventLogger: sourceInfo [periodCount=1, windowCount=1
D/EventLogger:   period [?]
D/EventLogger:   window [?, false, false]
D/EventLogger: ]
W/VideoCapabilities: Unrecognized profile/level 32768/2 for video/mp4v-es
W/VideoCapabilities: Unsupported mime video/mpeg2
D/EventLogger: loading [true]
I/art: Background sticky concurrent mark sweep GC freed 5201(231KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 2MB/2MB, paused 2.245ms total 127.757ms
I/VideoCapabilities: Unsupported profile 4 for video/mp4v-es
I/OMXClient: Using client-side OMX mux.
I/ACodec: Send reorder config(0) to VPU
E/ACodec: [OMX.Freescale.std.video_decoder.avc.v3.hw-based] storeMetaDataInBuffers failed w/ err -2147483648
W/ACodec: [OMX.Freescale.std.video_decoder.avc.v3.hw-based] prepareForAdaptivePlayback failed w/ err -2147483648
I/art: Background partial concurrent mark sweep GC freed 1907(84KB) AllocSpace objects, 1(64KB) LOS objects, 39% free, 2MB/4MB, paused 2.670ms total 150.068ms
I/art: Background sticky concurrent mark sweep GC freed 100(3KB) AllocSpace objects, 0(0B) LOS objects, 0% free, 4MB/4MB, paused 8.636ms total 105.842ms
D/EventLogger: sourceInfo [periodCount=1, windowCount=1
D/EventLogger:   period [?]
D/EventLogger:   window [?, false, false]
D/EventLogger: ]
D/EventLogger: Tracks [
D/EventLogger:   Renderer:0 [
D/EventLogger:     Group:0, adaptive_supported=N/A [
D/EventLogger:       [X] Track:0, id=3/49, mimeType=video/avc, res=1920x1080, supported=YES
D/EventLogger:     ]
D/EventLogger:   ]
D/EventLogger:   Renderer:1 [
D/EventLogger:     Group:0, adaptive_supported=N/A [
D/EventLogger:       [ ] Track:0, id=3/50, mimeType=audio/ac3, channels=6, sample_rate=48000, supported=NO_UNSUPPORTED_TYPE
D/EventLogger:     ]
D/EventLogger:   ]
D/EventLogger:   Renderer:2 [
D/EventLogger:     Group:0, adaptive_supported=N/A [
D/EventLogger:       [ ] Track:0, id=3/8241, mimeType=application/cea-608, language=eng, supported=YES
D/EventLogger:     ]
D/EventLogger:   ]
D/EventLogger: ]
D/EventLogger: videoEnabled [1.08]
D/EventLogger: videoDecoderInitialized [1.08, OMX.Freescale.std.video_decoder.avc.v3.hw-based]
D/EventLogger: videoFormatChanged [1.08, id=3/49, mimeType=video/avc, res=1920x1080]
I/art: Background partial concurrent mark sweep GC freed 509(23KB) AllocSpace objects, 0(0B) LOS objects, 39% free, 6MB/11MB, paused 3.876ms total 373.669ms
W/ACodec: using sliceHeight=1088 instead of what codec advertised (=0)
E/ACodec: dequeueBuffer failed.
D/EventLogger: renderedFirstFrame [Surface(name=null)/@0x226ffc18]
D/EventLogger: state [1.32, true, R]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [4.97, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [7.23, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [9.41, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [11.53, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [13.80, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [15.97, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [18.06, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [20.31, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: droppedFrames [22.44, 50]
D/EventLogger: loading [true]
D/EventLogger: loading [false]
D/EventLogger: loading [true]
Application terminated.

[rtamin]

I have some more information about why the video sample will not play on my device.

There is a check in MediaCodecVideoRenderer.java that drops the frame if the presentation time of the frame is lagging too much behind the current player's position.

The check is found here in MediaCodecVideoRenderer.java:

ExoPlayer/library/core/src/main/java/com/google/android/exoplayer2/video/MediaCodecVideoRenderer.java

Line 451 in 9f81485

if (shouldDropOutputBuffer(earlyUs, elapsedRealtimeUs)) {

It turns out for that video sample, after rendering a few frames, all subsequent frames that come out of the codec have a presentation timestamp that is lagging too much behind the current player's position. Also, as time goes by, I can see the presentation time of each frame keeps diverging more and more from the current player's position. So that is why the rendering of video stops after displaying a few frames.

I am actually working on 2 different android devices. Using the same ExoPlayer sample app and the same video sample, one of the devices can render the video without any problems. While the other device runs into the situation described above where the presentation timestamp keeps diverging away from the player's position.

Would anyone know what could be causing the presentation timestamp of the frames to be bad ?

Thanks

[rtamin]

I did some further investigation. This link has a diagram that summarizes the interaction of ExoPlayer with the codec to decode frames:

https://developer.android.com/reference/android/media/MediaCodec.html

On the left (input side) ExoPlayer feeds frames to the codec. On the right (output side) ExoPlayer reads decoded frames from the codec.

I find that on the input side, ExoPlayer is being slow to feed frames to the codec.

In my setup, I am running the experiment on my local network. The sample video is being served via http from a local http server on my local network, the URI is "http://192.168.100.5/movie.ts". I know the network connection/speed is not the issue here because serving other (even bigger and higher bitrate) videos to the same ExoPlayer have no issues to stream.

Would anyone have any thoughts about what could be causing ExoPlayer to be slow at feeding frames to the codec ?

[AquilesCanta]

I'll take a look into this as soon as I can. I haven't updated in some time, feel free to ping me.

[rtamin]

Thanks AquilesCanta.

I think the issue is the codec is slow at decoding the video frames.

The codec being used is: OMX.Freescale.std.video_decoder.avc.v3.hw-based

If the codec is slow, then the input side feeding frames into the codec will also take a long time to retrieve an input buffer from the codec. So that is why I was observing that ExoPlayer was being slow to feed frames to the codec.

I have identified the place where ExoPlayer requests for an input buffer from the codec:

ExoPlayer/library/core/src/main/java/com/google/android/exoplayer2/mediacodec/MediaCodecRenderer.java

Line 574 in 9f81485

inputIndex = codec.dequeueInputBuffer(0);

Reproduced below:

  inputIndex = codec.dequeueInputBuffer(0);
  if (inputIndex < 0) {
    return false;
  }

dequeueInputBuffer is being called with a timeout argument of '0', which means the call will not block and will return immediately.

I have added an integer counter to count the number of times ExoPlayer fails to receive a buffer. That number is pretty high on the device that cannot render the video. When I run the same experiment on a another device that renders the video, that counter is lower.

[AquilesCanta]

Have you by chance tried the platform player to see whether the observed behavior is similar?

[rtamin]

Hi AquilesCanta, by platform player you mean the Android MediaPlayer ?

I have tried Android MediaPlayer and MediaPlayer is able to play the video. Unfortunately, doing "abd logcat" while MediaPlayer is running, I cannot determine what driver MediaPlayer is using. All I see is MediaPlayer involves another player called OMXPlayer.

So maybe MediaPlayer is using a different codec than what ExoPlayer is using. OR, maybe MediaPlayer is configuring the codec differently than ExoPlayer. I noticed in MediaCodec.java there is a configure() method that passes a map containing some key=value configuration to the codec.

Link to configure() in MediaCodec.java
https://android.googlesource.com/platform/frameworks/base/+/android-5.0.2_r1/media/java/android/media/MediaCodec.java#545

[tresvecesseis]

In the past with some underpowered devices we had the same problem, in collaboration with the chip vendor we solved the problem enabling vsync (and hw deinterlacing), not allowing the MediaCodecVideoRenderer processOutputBuffer sleep on early frames or drop late ones and passing the positionUs and bufferIndex info to the codec (the device was on API19 and the codec reconfiguration and info passing were codec specific/propietary), with that changes the video was fluid again (13mbits h264 streams) so even with very underpowered hw you could have good result in exchange of some added latency if you have the help of the chipset vendor.

[ojw28]

We've made changes to do a better job of rendering video when the decoder can't keep up. These changes were first made in 2.6.0. Playback will still be choppy though. If the video decoder is too slow, that's a fairly fundamental problem that will prevent smooth playback.

Closing this issue, since it appears to be stale at this point.