Implement bitmap subtitle support

[perchrh]

Currently the ExoPlayer code assumes subtitles/text-tracks are always representable as text.

The metadata stream of the transport stream may contain SMPTE-TT data, an extension of TTML that extends TTML with binary blobs, encoded in base64.
The base64 data are part of the TTML xml structure and may for example contain bitmap subtitles or be used to tunnel other formats, like CEA-608.

My suggestion is to implement SMPTE-TT + PNG support, including the display of the base64-decoded bitmap subtitles in the player.

[ojw28]

What's your actual use case for this? You're not actually sending text as bitmaps, presumably?

[ened]

This seems to be a good summary: https://codesequoia.wordpress.com/2011/12/15/ttml_dfxp_smptett/

[perchrh]

@ojw28 The use case is support for DVB subtitles, as described in the article @ened links to.
The sequence of subtitle bitmap images is sent as a DVB transport stream along with the DVB video and audio, to be decoded by for example a set top box, before displayed on the TV.
Implementing this feature will make it easier to stream content created for DVB/television to Android devices. Transcoding the subtitle content may not be feasible, as the bitmap -> xml conversion cannot be done automatically.

[ojw28]

In the US at least, I don't think that delivering subtitles as bitmaps would meet FCC captioning requirements, which mandate that the user should be allowed to customize their visual appearance. Bitmap subtitles also wont scale well or look as nice as text rendered on the device, so it feels to me like this is something people should be looking to shift away from.

Do you agree with the above? This doesn't necessarily mean we wont add support. I'm just trying to gauge where this is actually useful, and for how long it will remain useful. Thanks!

[ened]

IIRC DVB streams are commonly found in taxi TVs here in Asia. The "subtitles" serve information like weather, news, pictures or ads as a picture overlay.

[perchrh]

@ojw28 Bitmap subtitles are suboptimal for many reasons, true. I wouldn't expect any new standard to go for that solution. My impression is that it's widely used, and I couldn't talk my broadcasting customers out of requiring it for their streaming solution. I could do some research to see how widely spread it is and if new solutions are being deployed or planned, it that's useful?

I've implemented this for a POC, using an older fork of ExoPlayer, by replacing the TTML parser[1], and using Android's BitmapFactory to create the PNG from the byte array from the base64 in the XML. An onBitmap callback was introduced in the player, akin to onText. The Subtitle/SubtitleHolder class could not be extended, though, and also had to be replaced.
[1] by using another open source parser with a permissive license

[IanDBird]

Just to add another point to this enhancement, but both DVD and BluRay disks use bitmap based subtitles (VOBSUB and PGS). Personally i'd love to see these handled from the Mp4Extractor and WebMExtractor's.

[perchrh]

With the latest ID3-parser fix, the XML data with the SMPTE-TT subtitles are sent to the player through the Id3MetadataListener interface. DemoPlayer#onId3Metadata listens for this event.
Perhaps a good place to implement this support is to modify the sender of this event, to instead parse the subtitles with ExoPlayer's TTML-parser (with additional base64->png/Bitmap-class code) and use the timing information from the TTML parser to send a signal over the SubtitleListener, so it will be received by DemoPlayer#onSubtitles. The interfaces may have to change a bit, to carry the Bitmap subtitles. Also, the Bitmap will have to be scaled for the current screen resolution (Surface dimensions).

[rvergara1]

Currently the ExoPlayer code assumes subtitles/text-tracks are always receive in text format (TTML).
Is it planned to implements SMPTE-TT + PNG support including the display of the base64-decoded bitmap subtitles in the player ?

[GoranNSW]

@ojw28 DVB standard is widespread in Europe and Asia. I agree that it is not the best solution for displaying subtitles, but standard is in use by majority of TV broadcasters.
Take a look into this document http://www.etsi.org/deliver/etsi_en/300700_300799/300743/01.03.01_60/en_300743v010301p.pdf where DVB subtitles are described in great detail.

[tresvecesseis]

I already submitted a patch adding dvb subtitle support, as we use it
internally we plan to submit it again for the v2 branch, we only considered
our use case but there are other examples of graphic subtitles, like the
DVD and Blu-ray ones, so maybe a discussion about a generic architecture
for graphic subs is needed.

El 1 ago. 2016 9:44 p. m., "GoranNSW" [email protected] escribió:

@ojw28 https:/ojw28 DVB standard is widespread in Europe
and Asia. I agree that it is not the best solution for displaying
subtitles, but standard is in use by majority of TV broadcasters.
Take a look into this document
http://www.etsi.org/deliver/etsi_en/300700_300799/300743/01.03.01_60/en_300743v010301p.pdf
where DVB subtitles are described in great detail.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#179 (comment),
or mute the thread
https:/notifications/unsubscribe-auth/AJ99vyXjyiOu2mL15IVgSfVg0jBod6s3ks5qbkyEgaJpZM4DCeZT
.

[GoranNSW]

Did you take into account the mpeg-ts stream?

[tresvecesseis]

Yep, our service is based on multicast rtp mp2t streams

From: GoranNSW [email protected]
Reply-To: google/ExoPlayer [email protected]
Date: Monday 1 August 2016 at 22:20
To: google/ExoPlayer [email protected]
Cc: "[email protected]" [email protected], Comment [email protected]
Subject: Re: [google/ExoPlayer] Implement bitmap subtitle support (#179)

Did you take into account the mpeg-ts stream?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[GoranNSW]

Thank you for the info. Looking forward to try it.

[tresvecesseis]

@GoranNSW I just issued a pull request #1781 adding support for DVB subtitles, I hope you will find it useful

[GoranNSW]

Thank you for your work and for letting me know. This is going to be more than useful, it is one of the main functions I need in my project. Thank you once again!

[tresvecesseis]

Let me know if it works for you, we have only tested in our streams and while we are pretty sure that the subs decoding are implemented correctly (at least for the one area case) the signaling in the PMT of the TS packet could be problematic, we are not sure if there are more ways to signal the DVB subtitles, in the past we have had problems with the AC3 audio streams being signaled differently depending of the contributor.

From: GoranNSW [email protected]
Reply-To: google/ExoPlayer [email protected]
Date: Wednesday 31 August 2016 at 01:37
To: google/ExoPlayer [email protected]
Cc: "[email protected]" [email protected], Comment [email protected]
Subject: Re: [google/ExoPlayer] Implement bitmap subtitle support (#179)

Thank you for your work and for letting me know. This is going to be more than useful, it is one of the main functions I need in my project. Thank you once again!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[GoranNSW]

I am sorry for such a long delay in response. I was working on one other project day and night.
Today I installed player v2 and tried with some streams but subtitles doesn't seem to work, they are not displayed in text selection dialog. Maybe they are not detected as a playable track?

Here are two samples with working DVB subtitles in VLC player, although exoplayer doesn't support them:

http://samples.ffmpeg.org/MPEG-VOB/transport/dvbsub/dvbsubtest.ts

https://bugs.launchpad.net/me-tv/+bug/624781/+attachment/1520761/+files/Tiededokumentti_%20Muistia%20etsim%C3%A4ss%C3%A4%20-%20YLE%20Teema%20-%20Wed%2025%20Aug%202010%2005_31_19%20PM%20EEST.mpeg

[tresvecesseis]

Have you tried the latest patches?, we have some problems with Android Studio without this:

ce2a80f

Also make sure that the fake eia-608 track is not activated

The only remaining problem we have is that SimpleExoPlayer reload the stream when you enable one subtitle track, also make

In any case, we will test your sample streams.

regards,

Sergio

From: GoranNSW [email protected]
Reply-To: google/ExoPlayer [email protected]
Date: Wednesday, 21 September 2016 at 04:28
To: google/ExoPlayer [email protected]
Cc: "[email protected]" [email protected], Comment [email protected]
Subject: Re: [google/ExoPlayer] Implement bitmap subtitle support (#179)

I am sorry for such a long delay in response. I was working on one other project day and night.
Today I installed player v2 and tried with some streams but subtitles doesn't seem to work, they are not displayed in text selection dialog. Maybe they are not detected as a playable track?

Here are two samples with working DVB subtitles in VLC player, although exoplayer doesn't support them:

http://samples.ffmpeg.org/MPEG-VOB/transport/dvbsub/dvbsubtest.ts

https://bugs.launchpad.net/me-tv/+bug/624781/+attachment/1520761/+files/Tiededokumentti_%20Muistia%20etsim%C3%A4ss%C3%A4%20-%20YLE%20Teema%20-%20Wed%2025%20Aug%202010%2005_31_19%20PM%20EEST.mpeg

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[tresvecesseis]

You have an ancillary page in your subs and that´s not common in the case of one subtitle service per PID, I have to check if it´s supported in our decoder.

From: "[email protected]" [email protected]
Date: Wednesday, 21 September 2016 at 11:41
To: google/ExoPlayer [email protected], google/ExoPlayer [email protected]
Cc: Comment [email protected]
Subject: Re: [google/ExoPlayer] Implement bitmap subtitle support (#179)

Have you tried the latest patches?, we have some problems with Android Studio without this:

ce2a80f

Also make sure that the fake eia-608 track is not activated

The only remaining problem we have is that SimpleExoPlayer reload the stream when you enable one subtitle track, also make

In any case, we will test your sample streams.

regards,

Sergio

From: GoranNSW [email protected]
Reply-To: google/ExoPlayer [email protected]
Date: Wednesday, 21 September 2016 at 04:28
To: google/ExoPlayer [email protected]
Cc: "[email protected]" [email protected], Comment [email protected]
Subject: Re: [google/ExoPlayer] Implement bitmap subtitle support (#179)

I am sorry for such a long delay in response. I was working on one other project day and night.
Today I installed player v2 and tried with some streams but subtitles doesn't seem to work, they are not displayed in text selection dialog. Maybe they are not detected as a playable track?

Here are two samples with working DVB subtitles in VLC player, although exoplayer doesn't support them:

http://samples.ffmpeg.org/MPEG-VOB/transport/dvbsub/dvbsubtest.ts

https://bugs.launchpad.net/me-tv/+bug/624781/+attachment/1520761/+files/Tiededokumentti_%20Muistia%20etsim%C3%A4ss%C3%A4%20-%20YLE%20Teema%20-%20Wed%2025%20Aug%202010%2005_31_19%20PM%20EEST.mpeg

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[GoranNSW]

Subtitles are working with your patches, great job! I have to wait and catch when subs are actually displayed in streams I am using, tracks are found and displayed in text selection dialog.

[GoranNSW]

I have finally succeeded to catch subtitles shown in the stream in VLC player, but in exoplayer track is not visible in selection list, so subtitle can't be displayed. I sent an email with the sample on [email protected]

Edit:

My mistake, I accidentally tried stream without your patch, now with your patch everything works excellent!

[tresvecesseis]

@GoranNSW thanks for your testing but I'm confused with your comments, are the subtitles working with the demo application and the #1781 pull request?

In any case. your published captures:

http://samples.ffmpeg.org/MPEG-VOB/transport/dvbsub/dvbsubtest.ts
https://bugs.launchpad.net/me-tv/+bug/624781/+attachment/1520761/+files/Tiededokumentti_%20Muistia%20etsim%C3%A4ss%C3%A4%20-%20YLE%20Teema%20-%20Wed%2025%20Aug%202010%2005_31_19%20PM%20EEST.mpeg

are too short, and the second one have malformed TS packets,.

Could you confirm if there are any remaining problems with DVB subtitles? Could you provide longer samples?

regards.

[GoranNSW]

I am sorry for making confusion, subtitles are working with demo app and pull request #1781.

Samples I earlier provided are found on the internet, and sample I sent an hour ago is actual sample I am using in my app. I can't post it here because of copyright material.

There are no problems with subtitles, if I find them I will report my findings.

[tresvecesseis]

I'm not related to google, my private email is [email protected]

El 23 sept. 2016 7:43 p. m., "GoranNSW" [email protected] escribió:

I am sorry for making confusion, subtitles are working with demo app
and pull request #1781 #1781.

Samples I earlier provided are found on the internet, and sample I sent an
hour ago is actual sample I am using in my app. I can't post it here
because of copyright material.

There are no problems with subtitles, if I find them I will report my
findings.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#179 (comment),
or mute the thread
https:/notifications/unsubscribe-auth/AJ99v6j9HqS902EnRoSqWWMGJwPIKJxvks5qtA_RgaJpZM4DCeZT
.

[GoranNSW]

I didn'k knew that, I sent you the sample if you want to test it

[tresvecesseis]

Thanks, we only have streams of our own service and we will appreciate any
samples you can provide.

El 23 sept. 2016 8:13 p. m., "GoranNSW" [email protected] escribió:

I didn'k knew that, I sent you the sample if you want to test it

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#179 (comment),
or mute the thread
https:/notifications/unsubscribe-auth/AJ99v1gq3jdxptHL69Q1fhlt92mUxMP0ks5qtBbbgaJpZM4DCeZT
.

[GoranNSW]

If you need more samples, for example with multiple audio tracks, I can send them to you.

[NikmanSergey]

@tresvecesseis, does ExoPlayer crew going to include your DVB subtitles support patch into official ExoPlayer v2 branch?

Could you, please, give some instructions how to apply this patch and then import this into existing project.
Are there already patched library download link?

[tresvecesseis]

It seems that the patch is not going to be integrated in exoplayer, they consider it a maintenance burden as the library is currently focused in OTT video support rather than IPTV or DTT, so they have put in place an mechanism to be able to implement custom elementary streams reader off-library and we are migrating our player to this API, you could do the same port with the published patch.

From: Sergey Nikitin [email protected]
Reply-To: google/ExoPlayer [email protected]
Date: Tuesday, 1 November 2016 at 10:02
To: google/ExoPlayer [email protected]
Cc: "[email protected]" [email protected], Mention [email protected]
Subject: Re: [google/ExoPlayer] Implement bitmap subtitle support (#179)

@tresvecesseis, does ExoPlayer crew going to include your DVB subtitles support patch into official ExoPlayer v2 branch?

Could you, please, give some instructions how to apply this patch and then import this into existing project.
Are there already patched library download link?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ojw28]

Hi. We've had bandwidth to look at the TV case in a little more detail recently, so it might be a good time to re-assess this (no promises of course). @AquilesCanta - What do you think? The pull request is here: #1781

@tresvecesseis - Regarding the pull request, can you confirm the code is OK with respect to whatever Google CLA you've signed here and if so remove references to other projects from it? Thanks!

[ojw28]

DVB support has been added in #1781 and cleaned up in 156bc52.

Is there anything else to do here, or can this issue be closed?

[ghost]

@ojw28

Issue: #1583 - was marked as a duplicate of this ticket. The issue was raised to cover support of SMPTE-TT in DASH.

From looking at the change log and testing the latest dev-branch, this does not seem to be supported (yet)

Maybe this could now be considered a secondary issue/feature?

[ojw28]

I'll re-open #1583 and close this one; thanks!