MP4Muxer and FragmentedMP4Muxer throwing illegal state exception at random times causing crash

[GhostCoder7]

Version

Media3 1.4.1

More version details

No response

Devices that reproduce the issue

Samsung Galaxy S20

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

I'm migrating to media3. I'm encountering an IllegalStateException when I use Mp4Muxer or FragmentedMP4Muxer to mux a video with two tracks - one video from the camera, one audio from the mic. I'm using the following format parameters:

video format parameters:
frame rate: 30
container mime type: MimeTypes.VIDEO_H264
sample mime type: MimeTypes.VIDEO_H264

audio format parameters:
channels: 1
sample rate: 44100
bit rate: 128000

Expected result

If you record a video before the IllegalStateException is thrown, the video records fine and can be played back - I've successfully recorded short 15second videos. It is expected that this would also be the case for long videos.

Actual result

If you record a video before the IllegalStateException is thrown, the video records fine and can be played back. But eventually after between 30 - 60 seconds, an IllegalStateException is thrown due to the following assertion and causes a crash:

MP4Writer.java (if using MP4Muxer):
...
private void flushPending(Track track) throws IOException {
    checkState(track.pendingSamplesByteBuffer.size() == track.pendingSamplesBufferInfo.size()); //<< triggered here
...

FragmentedMP4Writer.java (if using FragmentedMP4Muxer):
...
private ProcessedTrackInfo processTrack(int trackId, Track track) {
    checkState(track.pendingSamplesByteBuffer.size() == track.pendingSamplesBufferInfo.size()); //<< triggered here
...

It's not immediately clear why the sizes between pendingSamplesbyteBuffer and pendingSamplesBufferInfo would ever differ and therefore why the IllegalStateException would ever be thrown. There are no comments or documentation that explains why this would occur and how to avoid it.

Example stack trace:

java.lang.IllegalStateException at androidx.media3.common.util.Assertions.checkState(Assertions.java:85)
at androidx.media3.muxer.FragmentedMp4Writer.processTrack(FragmentedMp4Writer.java:296)
at androidx.media3.muxer.FragmentedMp4Writer.processAllTracks(FragmentedMp4Writer.java:289)
at androidx.media3.muxer.FragmentedMp4Writer.createFragment(FragmentedMp4Writer.java:239)
at androidx.media3.muxer.FragmentedMp4Writer.writeSampleData(FragmentedMp4Writer.java:126)
at androidx.media3.muxer.FragmentedMp4Muxer.writeSampleData(FragmentedMp4Muxer.java:185)
at xx.xx.encoder.VideoEncoderCore.drainVideoEncoder(VideoEncoderCore.java:374)
at xx.xx.TextureMovieEncoder.handleFrameAvailable(TextureMovieEncoder.java:1401)

Media

N/A

Bug Report

• You will email the zip file produced by adb bugreport to [email protected] after filing this issue.

[GhostCoder7]

Tried running the code again, and received this crash today (also at a random interval), possibly related:

java.util.ConcurrentModificationException
at java.util.ArrayDeque.nonNullElementAt(ArrayDeque.java:278)
at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:708)
at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:450)
at com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:555)
at com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:818)
at androidx.media3.muxer.FragmentedMp4Writer.processTrack(FragmentedMp4Writer.java:317)
at androidx.media3.muxer.FragmentedMp4Writer.processAllTracks(FragmentedMp4Writer.java:289)
at androidx.media3.muxer.FragmentedMp4Writer.createFragment(FragmentedMp4Writer.java:239)
at androidx.media3.muxer.FragmentedMp4Writer.writeSampleData(FragmentedMp4Writer.java:126)
at androidx.media3.muxer.FragmentedMp4Muxer.writeSampleData(FragmentedMp4Muxer.java:185)
at com.xx.xx.VideoEncoderCore.drainVideoEncoder(VideoEncoderCore.java:376)

I'm using a Handler for video frames, but not for audio frames.

[SheenaChhabra]

Hey @GhostCoder7 , The muxer APIs are expected to work on a single threaded model only. Since you are feeding audio and video from different threads, there could be some synchronisation issues. If you need to feed data from different threads then may be you could create some synchronised wrapper methods which then calls muxer APIs.