-
Notifications
You must be signed in to change notification settings - Fork 216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ASS/SSA Format: Introduce LayoutRes{X,Y} script headers to allow easy multi-resolution releases #641
Conversation
@TheOneric You can refer to Masaiki/xy-VSFilter@30afd3b to write GHA workflow, the key point is VSYASM and ExecutablePath |
Why not just add a function to Aegisub to adjust an existing sub for a new video? Just input the old and new res and let it do its magic to rescale the values in the script. If ISR is doing things wrong you need to open new tickets on my issue tracker for each individual issue. And a proper specification of how PlayResX/Y should behave when it differs from the stored/displayed video size would be useful as well. With practival examples. Same for LayoutRes. |
Thanks! I tried VSYASM before to no avail, but copying the |
Sorry for the delay, I didn't manage to finish this yesterday.
This would have the advantage of not requiring format additions, but means it will still not be possible to create self-contained ASS subs and still require author interaction and possible imprecision after one or multiple conversions.If I'm not mixing things up, writing such a script also isn't straightforward and no complete script (yet) exists. One place where ASS not being self-contained is an issue is e.g. streaming subs and videos, where the video stream may be downscaled according to available bandwidth. The received video storage size may not match the storage size the subs were authored with, so a naïve approach would result in incorrect rendering. It's possible (with libass and xy-SubFilter at least) to signal the original storage resolution via an additional channel, but that requires special handling not only on the server- but also client-side. With self-contained ASS regular streaming clients can be used (and only the server needs to e.g. on the fly insert appropriate So using new headers, avoids precision problems, requires no author-interaction, editors can even automatically add headers if none yet exists and allows for self-contained ASS subs simplifying other things. Since I believe issues from the addition to be minor to barely noticeable, I think these advantage justify the retroactive addition.
Sure, if you decide to implement compatible scaling, we can work out the transform matrices etc. Tbh, since it started quite a while ago, I couldn't give you the exact transformations otoh right now. To give some background and a brief overview though, there are three resolutions relevant to rendering conforming ASS:
Note, For classic guliverkli(2)-VSFilter and xy-VSFilter, rendering resolution is always equal to the video’s storage resolution, as they are only capable of rendering directly onto the directly decoded video frame, effectively reducing it to only two resolutions. xy-SubFilter and libass can render at arbitrary resolutions and because the exact storage resolution is required to keep rendering content the same, they receive the exact storage resolution from API-users. Rendering resolution usually is an isotropic scaling of the video’s native display resolution. Any choice of rendering resolution must result in an image visually quasi-identical to what the output would have been when rendering at storage resolution, just scaled accordingly. A valid pair of Here are some example files: layoutres_demo_01.tar.gz |
Indeed. No such tool exists yet, and when I worked on one for my personal use some years ago, it involved matrix decompositions, error-prone case-by-case logic, and the output required ugly ASS markup to achieve arbitrary horizontal offsets (which even seemed impossible at the time due to glitches in VSFilter’s rasterizer, but we’ve [re]discovered the (That’s for I may add that for some years already, some people have been making multi-resolution video releases with the exact same subtitle track attached, even though they would look bad in the most popular subtitle renderers, precisely because it would take effort to manually adjust the subtitles. Others do it because their entire workflow is automated (and no automatic rescaler exists). Ironically, some of those files look best in MPC-HC’s current internal subtitle renderer and in older VLC, which ignore(d) video storage resolution, and in fact some files accidentally produce heavy lag in the other renderers. So it might seem that making all renderers follow suit and ignore video resolution could be better. However, other files do rely on this video resolution dependence. (Indeed, that’s why libass ever started emulating it in the first place.) And among those multi-resolution releases I mentioned, I hear a big chunk actually need Thus, |
or mpv with the non-default |
Can you give a link to the ASS/SSA spec? The documents I find are not very detailed. Do I understand correctly that (for existing files) scaling should happen based on PlayRes vs. StorageRes, ignoring the anamorphic stretch of video (if any)? To be clean, I mean for rendering a sub that will be overlayed on top of the stretched video, not before stretching. And that you want to add LayoutRes to override StorageRes, so that same script will work no matter if the video is encoded anamorpic or not? Do encoders still use anamorphic in this day and age? Simple example: suppose that PlayRes is 360p, LayoutRes is 720, and I am watching/rendering at 1440p. Does everything need to be scaled by 4x, or are some there things that need to be scaled 2x? |
There isn’t one. The documents you’ve found are most likely the same ones that we have. We just have to emulate what older (more well-established) renderers do in practice. This isn’t just about anamorphic video, although that is certainly included.
LayoutRes would replace StorageRes in these calculations.
Assuming for the sake of the simple example that there’s no anamorphic stretching and ScaledBorderAndShadow is enabled, most things would be scaled 4x, but blur and the screen plane distance would be scaled only 2x. |
EDIT: Disregard this, astiob already tested XySubFilter. Sorry for the ping. Prebuilt DLLs can be found here as a GHA artificat: https:/TheOneric/xy-VSFilter/actions/runs/2941169396 or if you prefer you can use the source code to build it yourself. |
Suppose LayoutRes is 1920x1080 bundled with original 1920x1080 video. This script is then bundled with a version of the video where black bars are cropped away, giving for example 1920x800 video. Is that an allowed scenario? And if so, how should it be handled? If done as astiob says above it would mean a simple stretch of the rendered bitmap (LayoutRes -> DisplayRes), which would obviously give wrong AR in this scenario. |
If the display aspect ratio changes, it won't work and there is no general foolproof way this could be made to work. That's why the original post said (emphasis added):
What using LayoutRes also for PAR handling allows however, is to reuse scripts across different storage ratios and resolutions as long as the display aspect ratio stays the same. Consider e.g. subs originally typeset against a 16:9 PAL DVD release. DVDs must use anamorphic for 16:9 and in this case the next-best allowed storage resolution would be |
If you crop part of the video, you get a different video content. For a different video, you need different/adjusted subtitles. Put simply: when cropping the video, you must also crop the subtitles. There’s no way for a renderer to know that the video has specifically had black bars cropped (not to mention of some particular size). And the original subtitles on the original video may have had items positioned within those black bars; again there wouldn’t be any universally correct way for a renderer to decide what to do with them: a human must decide and edit the subtitles. This is the same regardless of LayoutRes, anyway. |
That is what I assumed. I just wanted to get it clarified so that it gets explicitly mentioned in the specification of the new header. |
1911bbd
to
cc35551
Compare
cc35551
to
d16d2b9
Compare
Rendering ASS depends on the video’s storage resolution for several tags. Thus transferring a subtitle file to a different version of a video, with e.g. higher resolution or anamorphic squeezing undone, requires adjusting all those tags. Affected are \be, \blur, \frx, \fry and if ScaledBorderAndShadow is not set to "yes", also all tags related to border and shadow. This locks all but simple subtitle files to a specific video storage resolution. If one wants to release several different resolution simultaneously, the same source is reencoded to undo anamorphic squeezing, or a new higher resolution source appears, it is required to manually adjust affected tags and each video version needs a different subtitle file. This is a pain point, and resulted in some releases just relying on user overrides of players, or incompatibly patched renderers to avoid the required manual adjustments. By adding new headers, which will replace the original video storage resolution in all calculations, authors will be able to create files which can be reused across different resolutions as long as the display aspect ratio stays the same. Hopefully this will also halt the spread of incomapatible patches and overrides, which just result in broken files and further ASS fragmentation. For simplicity and to avoid surprises, LayoutRes* headers only take effect if both are present and set to values larger than zero. If LayoutRes{X,Y} is set to corresponds to the actual storage resolution of the video the subs are authored and initially released with, at least the initial/main version will also be effectively compatible to older renderers, which do not understand the new headers yet. This will grant users some time to upgrade and minimise friction from this retroactive format addition. The header concept is also approved by some VSFilters and patches for Cyberbeing/xy-VSFilter’s xy-VSFilter and XySubFilter are pending with only implementation details being still up for discussion. Integration into active Aegisub forks and possibly other common editors will be pursued at a later date. libass-specific: Since API-users can initialise PAR, we must recalcute even existing values when LayoutRes{X,Y} is set to ensure the sub-author-provided values take precedence.
d16d2b9
to
1a533e5
Compare
Hi guys, I am working on LayoutRes implementation for MPC-HC ISR. Can you also make a conformance checking sample that uses LayoutRes, but without anamorpic stuff? And one with ScaledBorderAndShadow off. Thanks. |
Just a note: besides rotations, this is also known to affect
|
Some good simple demo files for those issues would be greatly appreciated. |
Sorry for the delay, I currently don’t have much time. For For |
Not in VSFilter. Or at least, I think this wording is more confusing than useful for VSFilter developers. We’re emulating code that VSFilter already has; it just needs to keep that same code but use a different scaling factor, moving the remaining scaling to a later time. But this reminds me that the |
I wasn’t sure if MPC-HC ISR still kept this behaviour, since other |
@Cyberbeing, @pinterf, @clsid2:
We would like to propose a retroactive addition to the ASS/SSA format, but in order to not fracture ASS even more we will only move forward with this if this extension will also be accepted and implemented in (most) active VSFilters.
This retroactive addition should be able to provide tangible benefits, while avoiding undue issues.
Some background first:
Rendering of the ASS/SSA format depends on the video’s storage resolution for several effects. This means subtitle releases are bound to a specific video (storage) resolution and in general cannot be reused for e.g. later BD-source releases of a higher resolution or in case of DVD-sources, different encodes which did/didn't undo the anamorphic stretching. (Simple, sparingly styled subs may happen to avoid this by not using the affected effects, but this isn't possible for more complex subs.) Affected tags are
\be
,\blur
,\frx
,\frz
and ifScaledBorderAndShadow
isn't set toyes
also all border and shadow tags.Thus we propose adding two new headers
LayoutResX
andLayoutResY
, which if both present and both set to legal (> 0
) values will be used in all places instead of the storage dimensions of the video file.This way, subtitle authors are able to create subs once and then reuse the very same file for future release with arbitrary video resolutions as long as the display aspect ratio matches the original (and timing + colours didn't change, but we can't do much about that and they are easier to semi-automatically adjust than the scaling-affected tags).
If the headers are not or only partially present or set to illegal values, rendering continues to work exactly as it always did before.
It seems fairly unlikely any existing subs put non-functional
LayoutRes{X,Y}
headers into their[Script Info]
section, so this change will not break existing subtitle files. By encouraging authors to set the layout resolution equal to the actual video storage resolution, the initial/main subtitle releases would also continue to work in older renderers, giving users some time to update their renderers instead of suddenly be confronted with bizarre rendering results.If this extensions is accepted, the plan is to also get common Aegisub forks to implement support for it and automatically initialise the headers as the actual storage resolution (and possibly, for the beginning at least, nudge users to keep it in sync with the actual resolution as currently happens for
PlayRes{X,Y}
)@Cyberbeing, @pinterf:
I wanted to draft up a patch implementing the headers for xy-VSFilter, but unfortunately I don't have access to any Windows dev setup and no familiarity with VS, so I can only offer this completely untested patch:
see here, or xy-VSFilter_layout-res_patches.tar.gz.
The patch is on top of Cyberbeing’s
xy_sub_filter_rc5
branch, specifically Cyberbeing/xy-VSFilter@fc01a8d though it likely applies to pinterf/xy-VSFilter as well after the incompatible scaling changes are reverted (see: pinterf/xy-VSFilter#34).I tried getting it to build in GitHub Actions, but this didn't work out either with the build failing to locate yasm no matter where I put
yasm.exe
. I'm willing to test and work on this patch, but would require assistance in getting a working build setup. Certainly welcome and ideal though, would be someone more familiar with both Microsoft VS and xy-VSFilter taking over the patches.@clsid2:
For MPC-HC’s Internal Subtitle Renderer the situation is actually a bit different from original guliverkli(2)-VSFilter, xy-VSFilter and libass. MPC-HC’s ISR is already incompatible with many existing files and other renderers for a long time (in our experience, this resulted in barely any complex subs targeting ISR). Among others, MPC-HC's ISR completely stopped considering the original video resolution during rendering. While this deviation has been around long enough for ISR to probably better keep it when no
LayoutRes{X,Y}
was set, it would be nice if the headers could be taken into account when they are present to reduce the incompatibilities to other softsub renderers.(Replaces #597 to get a new, backlog-free, discussion now that more people are involved and the previous points re.
LayoutRes{X,Y}
have all been decided.)