Large pixellation for some MPEG2 clips (using Android SW MPEG2 decoder) #2891
Comments
|
Sounds like you're manually enabling a decoder yourself in the platform, and observing that it doesn't work very well. Isn't that most likely to be an issue with the decoder? What decoder is it that's actually being used? There's nothing we can actually do to look at this issue given the lack of a bug report, device details and so on (edit: ignore previous comment about test content). |
|
I've sent the result of "adb bugreport' by email. The device is an Intel/Baytrail platform running Android-IA. The decoder is the one from Google, available in AOSP since marshmallow [1]. It used to be enabled by default [2], but for some reason this was changed on nougat [3]. The decoder is built into the firmware anyway, so I simply enabled it by including media_codecs_google_tv.xml into main media_codecs.xml. [1] https://android.googlesource.com/platform/frameworks/av/+/android-6.0.0_r1/media/libstagefright/codecs/mpeg2dec/ |
|
If it's a Google codec then you should probably file an issue on Android directly, as described here, under GfxMedia presumably. |
|
I was hoping you could inspect the sample clip to determine what special about it that can trigger pixellation, while the same decoder works fine with other clips. |
|
This is beyond what we realistically have time for, particularly given the decoder isn't enabled by default. Sorry. Please do file an issue on Android directly if you want someone to take a look. Thanks. |
|
I've made some experiments with ffmpeg, emulating the behavior of H262Reader. The strange thing was that I also got pixellation when using my experiment code, while ffplay could play the stream just fine. It appears that in the case of ffmpeg, the pixellation is due to the way the data is segmented and fed to the decoder. Based on H262Reader, I've segmented my data on picture start code boundaries (0x00 0x00 0x01 0x00). But it appears that when doing that, the decoder ignores any sequence and GOP headers that are embedded in the data chunks. My sample stream has additional sequence and GOP headers in the middle of the stream. These contains custom decoding parameters (quantisation matrix). If these are ignored by the decoder, decoding artefacts then appear. To make it work with ffmpeg, one must segment the data properly before feeding it to the decoder. In particular, the sequence and GOP headers must be isolated and fed separately. Given that similar pixellation problems appear when using the MPEG2 software codec from Google, I'd say it's possible it has the same cause, aka the way H262Reader segments the data. |
|
Does this imply you know how to fix it too ;)? Feel free to send a pull request if so. It's quite low priority for us, so it'll almost certainly be much faster for you to fix it. |
|
This (temporary/experimental) patch fixes the problem. The idea is to segment data at picture, sequence and GOP header boundaries. However, I ran into an exception while playing another sample clip that contains a discontinuity. The patch is available here: The exception is: |
|
I've noticed something unexpected regarding pixellation. If I change the container to mkv, keeping all elementary streams unchanged, then pixellation does not occur. I've changed container with: I've extracted the elementary video stream from both TS and MKV files and they're identical at the byte level. So I guess somehow data is segmented differently when using MKV. Any suggestion about where to look? Note: tested with this sample |
|
I've updated the patch for the branch dev-v2, you can find it here: I'm still getting an exception thrown when playing the test clip with the discontinuity: |
|
That failure implies your patch is writing incorrect data to the track output. For example, the values passed to output.sampleMetadata may imply the data of one sample overlaps that of another, or may refer to data that hasn't been written to the output yet. The intended segmentation of the stream into samples is that each sample should produce a corresponding output buffer after decoding. Is your patch a step toward achieving that, or is that what the implementation already does? And if that's what the implementation already does, is your patch segmenting such that there are more samples than we expect corresponding output buffers after decoding? |
|
The patch is intended to fix the segmentation of ExoPlayer, which is not suited to decoders from ffmpeg or Google. By not segmenting data into logical entities, the decoders miss important decoding parameters leading to pixellation and artefacts. |
|
That doesn't answer the question. The "correct" segmentation is that each input buffer to the decoder should correspond to a single output frame. The question I'm asking is whether your change achieves this, or whether this is already the case in the current version of your code, and your change is moving away from this. Which one of these is true? If the segmentation is already correct and the decoders are missing things, then we should get the decoders fixed and probably implement a re-segmentation workaround that's closer to where the decoder is used in the code-base, rather than being all the way up in an extractor that has no idea what decoder the samples are eventually going to be fed into. |
|
My patch was a deviation from the rule "1 sample -> 1 output buffer", which lead to the exception in case of discontinuities. Nevertheless, the segmentation in H262Reader is incorrect and I finally found the correct way that prevents the pixellation problem and does not break the above rule. Thanks. |
|
Ah, that's good news! Please let us know if/when you have an updated patch. |
|
@goffioul - Could you provide a pull request or explanation describing the correct segmentation? Thanks. |
|
@goffioul Thanks for the pull request. We'll review this and get it merged soon. As an aside, do you happen to know whether there is a guarantee that sequence header start codes always occur at the exact start of a packet, or whether it's ever possible for the start codes to straddle a packet boundary? I think our current implementation doesn't handle this case correctly, but I wanted to check if it can occur before submitting a fix. I'm not familiar with this format but it seems strange that |
|
@andrewlewis I'm not sure what you mean with "packet". Are you referring to the MPEG-TS packets? If yes, I think any start code could straddle across packet boundaries, and that's not the point of the proposed patch. The H262Reader class handles straddling start codes fine, I believe. The problem is how H262Reader segments the mpeg2video elementary stream into data chunk to be sent to the decoder. Especially when a frame sequence ends and a new one starts, the data stream would look like this: With FRAME_N being the last frame of the previous sequence, and FRAME_0 being the first frame of the next sequence (it's an I-frame). The part SEQ may contain new quantization factors that are applicable starting from FRAME_0. The existing H262Reader would segment the above data into 2 chunks/buffers, sent to the decoder: In the 2 decoders I've tested with (Android/Google's MPEG2 codec, and FFMPEG), the parts [SEQ] and [GOP] are ignored when they are appended like that to a frame. Those decoders expect [SEQ] and [GOP] to be prepended to the (first) frame they apply to instead. This then leads to incorrect decodings, as the quantization matrix from [SEQ] is not taken into account. The proposed patch segments the data into the following buffers instead: |
|
@goffioul Yes, I mean MPEG-TS packets. The issue is with how the I appreciate that your patch isn't trying to address this but noticed the problem in the existing code while reviewing it, and wanted to check if you knew whether there were guarantees about the positions of start codes. It sounds like there's no guarantee this situation can't arise, so I will make a fix. If you have test streams with lots of sequence headers it would be interesting to know if the sequence headers ever occur with their start codes at the end of a packet. |
|
@andrewlewis Got it, thanks. It's indeed a separate issue. I don't know whether there's a guarantee to have the sequence headers at the start of a packet. I have several sample clips that you may find useful: On a related note, I submitted another pull request (#3114) related to CEA-608 support in mpeg2/mpegts. The code is largely based on CsdBuffer, so it's possible the problem you're referring to also affects the pull request. |
Issue: #2891 ------------- Created by MOE: https:/google/moe MOE_MIGRATED_REVID=163951910
|
I made some further fixes to this. Please check everything is still working as you expect. Thanks! |
Issue: #2891 ------------- Created by MOE: https:/google/moe MOE_MIGRATED_REVID=163951910
I'm using a Intel-based device running Android-IA (Nougat), where I've enabled the MPEG2 codec in media_codecs.xml (SW-decoder available in Android since Marshmallow). My app is using ExoPlayer r2.4.1 (with ffmpeg extension built-in). I'm testing various MPEG clips using plain HTTP. Most of the MPEG2 clips play fine, but a few of them show massive pixellation. One sample is available at https://drive.google.com/open?id=0BwljeX6541LuamdieXZzamNkNFE . Those clips play fine in any other player.
As a comparison, here's a screenshot of how it looks like on Android:
https://drive.google.com/open?id=0BwljeX6541LuSW1BYW9YamlLeGM
Here's what the same looks like in VLC on Linux:
https://drive.google.com/open?id=0BwljeX6541LuM1B5ZnoycDNndms
Is the stream not compatible with ExoPlayer? Or with Android MPEG2 decoder? Is Android MPEG2 decoder simply doing a very poor job?
The text was updated successfully, but these errors were encountered: