Scaling the video independently of the size of the SurfaceView

[martinbonnin]

Is it somehow possible to manually control how the video is scaled in a SurfaceView ? What I have in mind is having something like a letterbox format but that could switch to VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING when the SurfaceView is collapsed.

I have read #1076 and https://code.google.com/p/android/issues/detail?id=37800 but none seem to provide a solution for letterboxing.

I guess I could workaround that by drawing myself in a GLSurfaceView using openGL but I'm unsure if this has battery/performance implications. Any idea how to do this ?

[martinbonnin]

Some more thoughts:

• For leterboxing, I can certainly use the LayoutParams and put a black view behind my SurfaceView or better, 2 black views on the sides to avoid overdrawing.
• VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING doesn't seem to be working on my Nexus 5X. No matter what I pass to MediaCodecVideoTrackRenderer constructor, the video is stretched to the exact size of the SurfaceView. Anyone seeing the same behaviour ?

[ojw28]

I don't think I really understand what the request is here. Please clarify?

To address the final point, I'm pretty sure VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING works correctly. Note that if you're using AspectRatioFrameLayout or similar then the scaling mode isn't going to make a difference; the difference will only be apparent if the SurfaceView has a different aspect ratio to the video. Note also that burnt in letter-boxing (i.e. black bars that are actually part of the video) count as part of the video w.r.t scaling modes.

[martinbonnin]

Hi ! Sorry for the confusion. There are 2 questions:

• I cannot get VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING to work. I modified the demo app to not use AspectRatioFrameLayout but I still get a stretched video either on a Nexus 5X or Nexus 5.
  • My modified version of the demo app is there: https:/martinbonnin/ExoPlayer/tree/test-VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING.
  • I get this result: https://drive.google.com/file/d/0B3dMoSG4-GHiTkJJOFRwbnpnVm8/view
  • Do I miss something ?
• Do we know what is the overhead of drawing the video to a GLSurfaceView with openGL instead of letting the MediaCodec draw directly to the SurfaceView ? Would you discourage this the same way using a TextureView is discouraged ?

[ojw28]

Nope, you didn't miss something. It appears that setVideoScalingMode doesn't work from the place where we're calling it. It should work, as far as I can tell by looking at the MediaCodec documentation. I'll follow up with the Android platform team about that.

I found that if you move the call to setVideoScalingMode in MediaCodecVideoTrackRenderer to happen in onOutputFormatChanged, then in that case the call does work correctly. It's probably the case that we should move the call there, seeing as that works, although I'd like to confirm expected behavior with the platform team first. In the meantime, you can give that a try at least and verify that it works as expected!

[ojw28]

Marking as a bug to make sure we fix/workaround application of the scaling mode.

[brianwernick]

If it helps either of you (or anyone else reaching this issue) I have implemented a Center_crop style (among other) scaling in brianwernick/ExoMedia#158 and fixed in https:/brianwernick/ExoMedia/pull/168/files.

In short I wait until the ExoPlayer informs us that the video size has changed, then wait for the TextureView to resize before applying the modified scaling

[martinbonnin]

Hi @brianwernick, I don't really want to use a TextureView unless you have significant data showing that it does not hurt the battery too much (see #1084).

I was considering using a GLSurfaceView instead which should be a bit better since it can still use the hardware layers for composition. But even that still requires some additional frames copy if I understand correctly.

Having VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING work would be the best solution, battery usage wise.

[brianwernick]

I agree that having the MediaCodec perform the work will be better performance/battery wise due to only having to do a single pass for each frame.

It does look like I skimmed over some information that was pertinent (that's what I get for looking at bugs after a full day of work). However worrying about the battery different between a SurfaceView and TextureView without data backing either assertion may be a little early; though I am interested in any data about that now :).

[ojw28]

• We'll push a fix to correctly apply the scaling mode soon, which will be to set it on the MediaCodec from onOutputFormatChanged as described above.
• TextureView is definitely significantly worse with respect to power draw. How much worse depends on many factors, but we've seen numbers in the region of a ~30% increase in overall device power draw when using TextureView rather than SurfaceView.

[brianwernick]

Thanks @ojw28.

[gbhall]

@ojw28 Any update? Thanks.

[gbhall]

Cheers @ojw28. Got this working using the dev branch library project. Any idea when you'll be pushing an updated master release? P.S. the applied fix doesn't seem to work if you're obtaining a Surface from TextureView. Cheers.

[ojw28]

Scaling mode only works for SurfaceView. We'll clarify this in the documentation. If you're using TextureView then you should use TextureView.setTransform instead.

[gbhall]

@ojw28 Cheers man. I'll stick to SurfaceView because I think I read from you that it comes with up to a 30% performance boost. However the solution doesn't seem to work for videos recorded off phones (tested Nexus 5 and LG G3). I've opened up another issue to easier track this: #1482. Cheers.