KHR_materials_basisu and independent data channels

[donmccurdy]

Opening a new thread here, based on discussion around #1612.

Storing three independent data channels in compressed textures introduces significant artifacts with current approaches (see #1612). As a result, these two cases in the current glTF material specification require consideration:

• occlusion/rough/metal (RGB) texture
• three-channel normal (RGB) texture

These could be solved with:

• (1) occlusion (R) and rough/metal (RG) textures, and
• (2) two-channel normal (RG) texture

A general-purpose texture-swizzling extension would be an alternative to (1). Currently that is harder to support on the web, but it won't always be so, and might be worth considering today.

Assuming (1) and (2) are the correct solutions here, what are the right mechanisms to bring them into the format? A couple ideas:

• (A) Introduce new extensions, e.g. KHR_packing_normal, KHR_packing_roughness_metallic, analogous to MSFT_packing_normalRoughnessMetallic.
• (B) Include them with the upcoming "PBR Next" material model's extensions; do not backport to existing metal/rough, spec/gloss, or unlit materials.
• (C) Include these changes, and perhaps other KTX2/Basis-related changes, in a glTF 2.1 release to simplify the perceived complexity, as compared to specifying multiple interdependent extensions.

[MarkCallow]

How would general-purpose texture-swizzling solve the occlusion/rough/metal case? In current compressed textures you can only have 2 independent channels: the RGB channel & alpha.

[lexaknyazev]

@donmccurdy
Possible channel mappings from Basis to GPU formats
KhronosGroup/KTX-Specification#100 (comment)

[donmccurdy]

@MarkCallow the format already allows occlusion to be separated from rough/metal, but even when they're separated rough/metal still points to the GB channels in the spec, the R channel isn't used. A texture swizzle extension would allow rough/metal to be read from different channels.

[zeux]

General-purpose swizzling doesn’t fully solve normal map storage - shader code is necessary to reconstruct the third component.

Two-component normal storage would be very welcome otherwise - high quality normal storage requires two separately compressed components, and games have been using DXT5 and, later, two-channel BC5, for >10 years now to solve this. However we should be careful wrt specification. On desktop the only widely supported format on the web that is suitable for high quality normals is DXT5 (BC3) but it requires storing one channel in G and one channel in A. If transcoding uses BC5 instead the mapping is RG. Not sure what the expected implementation strategy looks like.

How severe is the roughness-occlusion-metallic issue? Metallic is often binary and transition regions commonly have specular fringing due to interpolation even on uncompressed data. I have not tried to use BasisU extensively so don’t have a good intuition - would be good to have examples (uncompressed RGB, DXT1, BasisU transcoded DXT1).

[lexaknyazev]

I think the implementation strategy for the web is:

• EAC RG on mobile (WEBGL_compressed_texture_etc)
• BC5 on capable desktops (EXT_texture_compression_rgtc)
• BC3 on less capable desktops (WEBGL_compressed_texture_s3tc)

Basis Universal can be transcoded to all these formats.

Metallic-Roughness question came from customers who (at some point) had to use PNG over JPEG because of perceived artifacts. Basis Universal codec is not optimized for encoding three uncorrelated channels in the same image, so we might need to be proactive here.

[donmccurdy]

General-purpose swizzling doesn’t fully solve normal map storage...

Agreed, it only solves the occlusion/rough/metal problem.

How severe is the roughness-occlusion-metallic issue?

Good question – I don't know. It's been reported to us but I do think we should try to quantify it a little, maybe with something like https://googlewebcomponents.github.io/model-viewer/test/fidelity/results-viewer.html, before doing anything too elaborate to avoid it.

[zeux]

I think the implementation strategy for the web is:

Right, but then do you have to generate permutations for all shaders that need to read this data because you don't know the component mapping, and WebGL doesn't support general swizzles? General swizzles could be simulated with dot products I suppose, e.g. sample = textureFetch(), r = dot(sample, controlVector1), g = dot(sample, ControlVector2).

FWIW when I said that the only widely suported format on web/desktop is DXT5 (BC3) I meant that even though theoretically an extension is available, in practice Chrome doesn't implement it and neither does Edge (not sure about Firefox/Safari).

[lexaknyazev]

permutations for all shaders

Well, that depends. EAC RG11 and BC5 are sampled from red and green; BC3 (DXT5) would use red and alpha. In my opinion, the former should be the default.

BC4 and BC5 formats are mandatory on all D3D10 and newer class hardware, so they should be widely available on desktops. The corresponding WebGL extension is available today in Firefox when using OpenGL. Hopefully, ANGLE will expose it soon as well. This would unlock Chromium-based browsers including Edge Insider.

ETC2/EAC formats are available today in Chrome and Firefox on Android with capable (ES 3.0+) hardware.

Safari will catch up someday...

[donmccurdy]

Any idea what percentage of desktop devices support EXT_texture_compression_rgtc? I was hoping to find that on https://webglstats.com/webgl/extension/WEBGL_compressed_texture_s3tc but didn't. :/

[lexaknyazev]

Oldest desktop hardware that support RGTC is about 12 years old:

• AMD Radeon HD 2000
• Intel GMA 4500
• NVIDIA GeForce 8

[zeux]

Any idea what percentage of desktop devices support EXT_texture_compression_rgtc?

Almost all GPUs support the formats, but almost no browsers do. WebGL commonly works through ANGLE which, when targeting DX9, can't implement this extension because the format is DX10-only; I'm not sure what the status is for ANGLE-DX11.

[MarkCallow]

WebGL commonly works through ANGLE which, when targeting DX9, can't implement this extension because the format is DX10-only; I'm not sure what the status is for ANGLE-DX11.

This kind if thing is precisely why all compressed formats are expressed as extensions in WebGL. In other words just because ANGLE/DX9 can't support RGTC is no reason for browsers to not support it when running on DX10 or above. I am in communication with the author of webglstats.com to find out why it doesn't list EXT_texture_compression_rgtc among the extensions. I.e, is it because it no browser has ever reported it or because it doesn't know about it.

[lexaknyazev]

ANGLE support for RGTC can now be tracked here:
https://bugs.chromium.org/p/angleproject/issues/detail?id=3149

Chromium support for RGTC (and BPTC) here:
https://bugs.chromium.org/p/chromium/issues/detail?id=1013369

[zeux]

Just so that I understand, is this issue separate from the KTX2 PR (aka with the KTX2 PR we'll only get support for three-channel normals)?

Experimenting with Basis and while I like the quality of diffuse textures, normal quality is not very good. This is using normal map encoding mode but it didn't help much. I haven't tried using 2-channel normal maps with BasisU yet - not sure if the idea is to use 2 channel data with the existing ETC1S encoding, or to have two channels individually stored as ETC1S streams.

[MarkCallow]

Just so that I understand, is this issue separate from the KTX2 PR (aka with the KTX2 PR we'll only get support for three-channel normals)?

Which KTX2 PR are you referring to?

not sure if the idea is to use 2 channel data with the existing ETC1S encoding, or to have two channels individually stored as ETC1S streams.

A 2 component texture is supposed to be encoded with R in all 3 components of one ETC1S stream and G in all 3 components of a second. This is what toktx/libktx and basisu_tool do when the input image has just 2 components. The separateRGToRGB_A option and its equivalent in basisu_tool are only needed when the input is a 3 or 4 component image. You should specify normalMap in both cases, if the data is a normal map.

Whether splitting the components across 2 ETC1S streams in this way improves quality, I don't know but @richgel clearly thinks it does as he has implemented this feature.

[zeux]

Sorry - referring to #1612.

I think I get it but also am slightly confused. It sounds like there are two options:

1. Split two channel image into two RGB streams
2. Split two channel image into RGB and A stream

It seems like the "separate RG" option in basisu does the latter? Or are they the same in that alpha is also encoded as ETC1S?

[zeux]

Using seperate_rg_to_color_alpha option of basisu indeed produces much better quality - it needs support for DXT5 (BC3) decompression of course, and shader changes to accomodate two-channel normal maps. The screenshot here is from a modified copy of three.js where I hacked it in.

[lexaknyazev]

@zeux
See this section of the KTX2 spec for the description of Basis Universal channels storage and mapping.

[MarkCallow]

@zeux, if you've looked at the reference @lexaknyazev provided you will have seen that alpha is indeed encoded as ETC1S so your 1 & 2 are the same.

Separating R & G looks a lot better. I wonder how much of this is due to separating them for compression and how much is due to using DXT5 as the transcode target. It is beyond my current knowledge.

What is the format of the input images you are using. Is it a 3 component normal map so you just omit the 3rd component? Is there no need to specially generate a 2-component normal map as the input?

[zeux]

@MarkCallow I haven't done the experiments to try to prove it or disprove it but I suspect that the limiting factor is ETC1S so splitting into two channels helps for this reason. This is based on the fact that I'd expect BC1 encoded normals to not look quite as bad as the screenshots above indicate (edit and also I think ETC1S is in general noticeably weaker than BC1 so I'd be surprised if BC1 itself was a limiting factor). I will try a direct BC1 encoding using something like nvtt and upload another screenshot just so that we have a data point.

And yeah, it's sufficient to simply omit the third component, so on the encoding side the only flag that's necessary is seperate_rg_to_color_alpha. This is because in tangent-space normal map storage, it's common to assume that .Z is non-negative (otherwise normals are encoded in the negative hemisphere from the vertex normal's halfspace point of view, which commonly isn't required). So .z can be reconstructed from .rg or (in this case) .ga components depending on the hardware format used for encoding. The reconstruction is simply sqrt(max(0.0, 1.0 - dot(n.xy, n.xy))). This is the same approach that is commonly used for 2-channel normal map encoding when targeting specific hardware formats like BC3 or BC5.

[zeux]

Here's the DXT1 encoded normal map using nvtt, with all other textures encoded using Basis. Note that the quality is very good compared to either Basis options - Basis substantially distorts the normals whereas DXT1 by itself doesn't. I will experiment with different Basis quality settings but it looks like the issue isn't with the transcode target format. update tried using Basis with -q 255 -comp_level 5 - encoding took forever (45 minutes for 12 2K textures...) but the normal quality isn't noticeably better vs the encoding with default settings I posted above.

[lexaknyazev]

@zeux
Basis Universal internally uses "ETC1S" encoding - a subset of ETC1 compressed texture format that can be easily transcoded to DXT-style formats.

To put it simply, an ETC1S RGB block (4x4) has a local palette of 4 colors that are located on a straight line segment that is parallel to the main diagonal (0, 0, 0) -> (255, 255, 255).

The complete description of ETC1 (and ETC1S) can be found in the KDFS v1.3.

For comparison, a DXT RGB block has a local palette of 4 colors located on a straight line segment that can be oriented in any direction (within quantization limits).

When Basis Universal encoder is used with 2-component image, it puts each channel into a separate ETC1S slice. At runtime, these two slices need to be transcoded to

• two ETC1 textures (ES 2.0)
• EAC RG11 (ES 3.0+)
• BC5 (D3D10+)
• BC3, with the second source channel going into the alpha target channel (D3D9+)

For WebGL, BC4/BC5 formats are provided by the EXT_texture_compression_rgtc extension.
It is already available in Firefox with ANGLE disabled (see webgl.disable-angle in about:config).

Chromium/ANGLE implementations can be tracked here:
https://bugs.chromium.org/p/angleproject/issues/detail?id=3149
https://bugs.chromium.org/p/chromium/issues/detail?id=1013369

[aras-p]

FWIW, shader code needed in order to support both BC5 (RG components store two normal map channels) and DXT5nm (AG components store two normal map channels) as used in Unity, is somewhat cheaper than two additional dot products, in fact just one additional mul:

// Unpack normal as DXT5nm (1, y, 1, x) or BC5 (x, y, 0, 1)
fixed3 UnpackNormalmapRGorAG(fixed4 packednormal)
{
    packednormal.x *= packednormal.w;

    // reconstruct Z
    fixed3 normal;
    normal.xy = packednormal.xy * 2 - 1;
    normal.z = sqrt(1 - saturate(dot(normal.xy, normal.xy)));
    return normal;
}

This does require the DXT5nm style encoder to put 1.0 into the "unused" red & blue channels.

[zeux]

@aras-p Thanks, this is a neat trick I forgot I knew. Just to clarify, it’s only essential that red channel contains 1 - blue could be an arbitrary value including 0?

[zeux]

@lexaknyazev Thanks, I wasn’t aware that ETC1S excludes support for deltas. This is unfortunate, as it makes it even weaker than ETC1 (I understand that this is a compromise to find a common ground between ETC1 and DXT1) which is already not as good as DXT1, and is unlikely to be a good fit for three component normals. I am wondering if two component normals stored in a single ETC1S slice would fare better, will test this later.

update no, this doesn’t seem to help. I guess the only solution is to use two separate ETC1S slices.

With ETC1S used to encode x/y separately I wouldn’t expect a dramatic difference in quality between BC2 and BC5 - ETC1S should be the quality bottleneck - so it sounds like the main issue is specifying this such that the renderers can use two channel normals, including a possible variance in supported swizzles, and this will work as well as possible within constraints of BasisU even without browser support for BC5.

[aras-p]

Just to clarify, it’s only essential that red channel contains 1

Yeah.

[lexaknyazev]

With ETC1S used to encode x/y separately I wouldn’t expect a dramatic difference in quality between BC2 and BC5

BC2 stores alpha explicitly, quantized to 4 bpp. Did you mean BC3?

• BC3 alpha block is exactly the same as one channel of BC4/5.
• Single-channel ETC1S transcoded to BC3 RGB should be slightly worse than transcoded to BC4/5.

[zeux]

Yes - I meant BC3.

[MarkCallow]

I can see very little difference between the -separate_rg_to_color_alpha version and the custom (255,Y,255,X) encoder version. If anything the former is slightly better.

[zeux]

@zeus, when you did your tests with FlightHelmet did you use set normalMap in the Basis encoder parameters (or use the equivalent CL options)?

Yes - all screenshots above are using -normal_map.

I can see very little difference between the -separate_rg_to_color_alpha version and the custom (255,Y,255,X) encoder version. If anything the former is slightly better.

Right, I'm expecting separate_rg_to_color_alpha to result in slightly higher quality - the reason why (255, Y, 255, X) option seems attractive is that it allows us to push the work required to keep normal map decoding branchless into encoder without having to implement transcoder-side support for overriding channels for all output formats like BC3, ETC2, etc.

[lexaknyazev]

RGTC (BC4 and BC5) support has landed in ANGLE.

Follow https://bugs.chromium.org/p/chromium/issues/detail?id=1013369 to see when it comes to Chromium.

RGTC support has landed in Chrome Canary.

[richgel999]

FWIW - UASTC support will be landing in Basis Universal very soon. This will give developers a very high quality alternative for normal maps vs. ETC1S. However, the tradeoff will be larger files. The same low-level transcoder will nearly transparently support both ETC1S and UASTC textures.

The initial release of UASTC will be focusing on functionality and highest quality first, with compression ratios taking back seat. Over time we can improve the RDO encoder.

[donmccurdy]

A quick set of tests on our WaterBottle sample are given below. The first screenshot shows the model with its original uncompressed textures. Each row after that includes only a single Basis Universal texture, the rest are uncompressed. All use toktx --genmipmap --bcmp --clevel 5 --qlevel 255 (highest quality levels for ETC1S?), as well as --normal_map for the normalTexture example to disable selector and endpoint RDO.

slot	screenshot
original
normalTexture
metallicRoughnessTexture
baseColorTexture

There's nothing new with the pixelated/blocky effect on the normalTexture — I haven't separated the X/Y channels here, so that's an expected issue.

The artifacts at the edges of the metal/nonmetal logo edges for the metallicRoughnessTexture sound like what @zeux suggested earlier:

Metallic is often binary and transition regions commonly have specular fringing due to interpolation even on uncompressed data.

I don't have any immediate takeaway here, except that this provides some early support for a not particularly unexpected conclusion: we'll need to be pretty careful about texture packing. I realize UASTC is available for higher quality results, but we should try to provide the best recommendations we can for users who need the performance aspects of ETC1S.

Screenshots created with toktx, donmccurdy/glTF-Transform#36, and https://gltf-viewer-experimental.donmccurdy.com/.

[donmccurdy]

A clearer picture of the metallicRoughnessTexture issue:

before	after

[MarkCallow]

I realize UASTC is available for higher quality results

Even with UASTC it may be better qualitywise to separate the X & Y or metallic & roughness into the RGB and A components. As I understand it, i.e. been told, the current spec. requires they be in the RGB components. Therefore a new glTF material spec will be needed that permits separation.

Btw don't get hung up on 1 or 2 slices. That is invisible except in the case where no RGBA format is available and an engine wants to transcode RGB & A into 2 separate textures. Even then it is not about there being 2 slices but about there being alpha data. You'd have to do the same with a UASTC based texture that has alpha. We can avoid this by saying that if the GPU does not support any RGBA block-compressed format, the engine should transcode to uncompressed RGBA32. The important thing is to allow separation.

[donmccurdy]

Therefore a new glTF material spec will be needed that permits separation.

Here's what I believe it would take to optimize glTF material textures for ETC1S:

• add normalTextureMode: "RGB" | "RA" property (allowing both normal map modes)
• add metallicTexture B and roughnessTexture G properties
• deprecate metallicRoughnessTexture, or continue supporting it
• (Optional) Create a KHR_texture_swizzle extension to allow more flexibility in the "fewer samplers vs. fewer compression artifacts" tradeoffs for end users.

Setting aside the complexity of those changes, whether they'd be extensions or glTF 2.1, etc... would these actually solve the problem? Do we need more testing to be confident of that? These aren't quick fixes, but I'm worried about releasing the extension in a state where ETC1S can only be used for half of glTF's textures.

[lexaknyazev]

add metallicTexture G and roughnessTexture B properties

Why swapping channels there? The core spec samples metalness from B.

[donmccurdy]

Oops, thanks — that was not intentional, I'll fix the comment.

[lexaknyazev]

So the author would have to choose one of (assuming BasisU is used, in the order of increasing quality):

• ETC1S RGB-ORM
• ETC1S R-O + ETC1S GB-RM
• ETC1S R-O + ETC1S G-R (new) + ETC1S B-M (new)

My main concern here is that the runtime would have no flexibility as the compressed textures cannot be merged to reduce the number of samplers.

[lexaknyazev]

would these actually solve the problem?

The third option (with 3 independent ETC1S textures for ORM) is the best what could be achieved with ETC1S. Further quality increase would require switching over to UASTC.

UASTC can preserve 2 fully-independent channels (actually it's 3 + 1, where 1 can be any of RGBA) but only when transcoding to ASTC or BC7.

[donmccurdy]

I'm not sure I understand the syntax you're using to describe the options, sorry. What are R-O, G-R, and B-M?

[lexaknyazev]

Occlusion from Red; Roughness form Green; Metalness from Blue.

[donmccurdy]

Ok — yes, I agree those are the options, given these hypothetical spec changes. Note that they're ordered not just by increasing quality, but by increasing sampler count. I think of sampler count as an optimization to be done at authoring time. So in that sense, it's "good" that we preserve options on both ends of the spectrum: packing all three into one RGB texture (it certainly works for PNG, and might be okay sometimes with ETC1S too!) while making full quality with three slices available.

Or at least, that's the best we can do without (a) requiring that certain data always use alpha channels, or (b) supporting arbitrary swizzle. I really don't want to do (a) retroactively. I could be convinced on (b), as a standalone future extension — that seems like the only futureproof way to optimize both sampler counts and texture quality at the same time, to me.

[lexaknyazev]

No worries - IBL standardization is on the roadmap. As different texture types present different challenges wrt GPU compression, we decided to tackle them one at a time.

This issue is specifically for material textures containing non-color data.

[lexaknyazev]

This issue is about material textures with independent channels, like occlusion/roughness/metallic data. Special-encoding formats like RGBM/RGBE that require cross-channel evaluation will be handled separately.

[donmccurdy]

@polarathene thanks for the comments! You're just several steps ahead of what we're working on in this thread haha. Perhaps you could start a new issue on KTX and RGBM, so we don't lose that input. 🙂

[donmccurdy]

If anyone would like to test different settings with their own glTF models, I've published an authoring implementation of KHR_texture_basisu in @gltf-transform/cli:

# Installation
npm install --global @gltf-transform/cli

# Compress normal textures with UASTC, everything else with ETC1S
gltf-transform uastc input.glb output.glb --slots "normalTexture"
gltf-transform etc1s input.glb output.glb --slots "!normalTexture"

I'd recommend testing files in https://sandbox.babylonjs.com/ rather than my viewer, which is still on an experimental branch and lacks ZSTD decoding support.

[donmccurdy]

It seems like we've reached consensus that regardless of the challenges involved in packing independent channels into a single texture, the KHR_texture_basisu extension should not be concerned with defining alternative packing of normal maps or other textures. We can recommend that users upgrade to UASTC when ETC1S results are insufficient, immediately, and may consider the changes in #1682 (comment) in the future, as separate extensions or a glTF version.

We will also need to be careful with channel decisions on the upcoming PBR next (transmission, volume, sheen, ...) texture types.

Since the original questions of this thread are resolved and this doesn't block finalization of KHR_texture_basisu, I'll close this issue. Thanks, all!