trun from Text track is being associated with wrong TrackBundle when playing video with both a Video track and a Text track

[waltojac]

When attempting to play a fragmented .mp4 video that contains a Video track and a Text track (see mp4info screen grab below), the trun from the Text track is being added to the Video track causing an "Invalid NAL length" ParserException to be thrown. I believe this is caused by the fix added for this issue in FragmentedMp4Extractor.java on line 921.

You can reproduce this by playing the video I emailed to [email protected] in the sample exoplayer app. That video plays in all other video players I have tried including Chrome, Firefox, iOS, and more.

mp4info dump from Bento:

> mp4info sampleVideoWithTextTrack.mp4
File:
  major brand:      avc1
  minor version:    0
  compatible brand: avc1
  compatible brand: isom
  fast start:       yes

Movie:
  duration:   0 ms
  time scale: 27500
  fragments:  yes

Found 2 Tracks
Track 1:
  flags:        7 ENABLED IN-MOVIE IN-PREVIEW
  id:           1
  type:         Video
  duration: 0 ms
  language: ```
  media:
    sample count: 0
    timescale:    27500
    duration:     0 (media timescale units)
    duration:     0 (ms)
    bitrate (computed): 2900.260 Kbps
    sample count with fragments: 1716
    duration with fragments:     1716000
    duration with fragments:     62400 (ms)
  display width:  1280.000000
  display height: 960.000000
  Sample Description 0
    Coding:      avc1 (H.264)
    Width:       1280
    Height:      960
    Depth:       24
    AVC Profile:          77 (Main)
    AVC Profile Compat:   0
    AVC Level:            42
    AVC NALU Length Size: 4
    AVC SPS: [274d002a9a6402803cd80b7010101400000fa000035b63a18007320000731e2ef2e343000e640000e63c5de5c280]
    AVC PPS: [28ee3c80]
    Codecs String: avc1.4D002A
Track 2:
  flags:        7 ENABLED IN-MOVIE IN-PREVIEW
  id:           2
  type:         Text
  duration: 0 ms
  language: ```
  media:
    sample count: 0
    timescale:    27500
    duration:     0 (media timescale units)
    duration:     0 (ms)
    bitrate (computed): 3.852 Kbps
    sample count with fragments: 578
    duration with fragments:     1716660
    duration with fragments:     62424 (ms)
  Sample Description 0
    Coding:      text

Error Stack Trace:

E/ExoPlayerImplInternal: Playback error
      com.google.android.exoplayer2.ExoPlaybackException: Source error
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:580)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:223)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: com.google.android.exoplayer2.ParserException: Invalid NAL length
        at com.google.android.exoplayer2.extractor.mp4.FragmentedMp4Extractor.readSample(FragmentedMp4Extractor.java:1365)
        at com.google.android.exoplayer2.extractor.mp4.FragmentedMp4Extractor.read(FragmentedMp4Extractor.java:340)
        at com.google.android.exoplayer2.source.BundledExtractorsAdapter.read(BundledExtractorsAdapter.java:127)
        at com.google.android.exoplayer2.source.ProgressiveMediaPeriod$ExtractingLoadable.load(ProgressiveMediaPeriod.java:1048)
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:415)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

Exoplayer version: 2.13.3
Android version: 11
Android device: Pixel 3 XL

[ojw28]

I'm not really sure why we committed a workaround for the issue you refer to. I'm tempted just to revert it.

[ojw28]

That said, I'm also not sure what the text track in the attached file is. Have you found any player that will actually display the text track, and do you know how this file was created?

[waltojac]

Thanks for getting back so soon. Agreed, doesn't seem quite right. I am not interested in displaying the text track in a video player, it is just a way to associate metadata from our custom hardware with the video that we recorded. We store are going to store the data as JSON in the text track to be parsed later. This video was made by us using custom hardware.

[ojw28]

Got it. The workaround that introduced the problem you're seeing is:

If there is only one track, we can assume that both boxes refer to the same track even if the track indices don't match.

The problem with it is that "only one track" is actually implemented as "only one track that we recognize", which is not quite the same thing. In your video there are two tracks, but we don't recognize the text track you're adding. To fix this we should either revert the workaround or plumb through the unrecognized tracks (or at least a boolean indicating that there are some) somehow.