Improving texture support for 1.0.1+

[lexaknyazev]

List of possible issues worth considering

• Cube mapping support cube mapping proposal #266
• Uncompressed formats support (points copied from Thoughts on tightening up the spec #563)
  • Uncompressed is the simplest possible pixel format. All GL implementations support it. It looks like texture.type has no use without KHR_binary_glTF while spec & extension are in their current state.
  • Advanced use cases for this: procedurally generated at runtime materials, FLOAT textures with precomputed data. Enabling OES_texture_float & OES_texture_half_float could be done similar to Draft of uint32 support #619.
  • (original proposal) Add optional width, height and mimeType fields to image and remove them from KHR_binary_glTF\image. This will serve two purposes:
    1. We'll be able to use uncompressed textures both from external files (with explicit width, height and "mimeType": "application/octet-stream") and from embedded data: URI. In the latter case, we should omit mimeType field and use URI: data:application/octet-stream;base64,....
    2. KHR_binary_glTF\image will contain only bufferView just like KHR_binary_glTF\shader.
  • Adding fields to image is both ways compatible, removing fields from KHR_binary_glTF\image is not. New/updated assets won't work with older versions of Cesium, so changes to KHR_binary_glTF could be queued up for the next version to preserve compatibility.
• Compressed textures support Add compressed textures to spec #59
  • We need to consider multiple versions of the same texture for different compression formats (it's up to runtime to select one from RAW/PVRTC/ETC/S3TC/JPG). Export/conversion tools can provide only uncompressed file while all others can be generated during pipeline processing.
  • Spec can define some sort of framework for adding texture formats in future via extensions.
• Multiple resolutions of the same texture for high/mid/low hardware. Again, this processing can be done by pipeline tools.
• EXT_texture_filter_anisotropic

[pjcozzi]

To control the scope, I suggest

• Cube mapping support cube mapping proposal #266

Not considering this until PBR.

• EXT_texture_filter_anisotropic

Not considering this until this is enough community interest.

• Multiple resolutions of the same texture for high/mid/low hardware. Again, this processing can be done by pipeline tools.

Isn't this implicit in the ability to have different materials, each of which can have different parameters, e.g., with different resolution textures?

---

For the remaining items, it would be good if you can provide example JSON snippets for discussion, then we can iterate and formalize them in the schema and spec.

The scope may turn out to be too big for 1.0.1; keep in mind that we need to actually implement these features as well to make sure they are pragmatic and solid enough for inclusion.

---

In general, we need to be very careful about adding too many features to glTF too soon. The community needs a reliable, consistent format that is widely supported by exporters and easy to implement importers. If we add too much complexity too early in glTF's life, we risk making the barrier-to-entry too high, and therefore, hurt adoption and consistency of implementations.

[lexaknyazev]

Cube mapping support #266

Not considering this until PBR.

It depends on how we treat glTF: scene format vs asset format. I think it's perfectly fine to have, e.g., simple skybox without any PBR since cubemaps are in the core OpenGL ES 2.0 spec.

Multiple resolutions of the same texture for high/mid/low hardware. Again, this processing can be done by pipeline tools.

Isn't this implicit in the ability to have different materials, each of which can have different parameters, e.g., with different resolution textures?

I don't think so, because mesh primitive can reference only one material. So it would be an application specific approach to switch them. Anyway it's a really complex issue (meshes LoD, textures resolution/format, shaders with optional code paths via conditional preprocessor directives).

I think the real question is about the glTF's mission: universal runtime format that can be used with (almost) any engine (like image/sound/video file formats as it was mentioned in glTF slides) or some sort of "building block" for a particular application. It can be something in between, of course.

For the remaining items, it would be good if you can provide example JSON snippets for discussion, then we can iterate and formalize them in the schema and spec.

Already started working on them.

In general, we need to be very careful about adding too many features to glTF too soon. The community needs a reliable, consistent format that is widely supported by exporters and easy to implement importers. If we add too much complexity too early in glTF's life, we risk making the barrier-to-entry too high, and therefore, hurt adoption and consistency of implementations.

I believe it's kind of a chicken-egg problem. If format doesn't provide enough functionality why should someone use glTF instead of creating a proprietary format?
Most of mentioned issues are legitimate parts of OpenGL ES 2.0 / WebGL 1.0 spec with widely supported extensions.
It's unquestionable that we have to be careful and think about forward and backwards compatibility.

[pjcozzi]

Cube mapping support #266

Not considering this until PBR.

It depends on how we treat glTF: scene format vs asset format. I think it's perfectly fine to have, e.g., simple skybox without any PBR since cubemaps are in the core OpenGL ES 2.0 spec.

Agreed. It's just a matter of 1.0.1 spec scope and priorities. The goal of the 1.0.1 is to aid in validation; we need to be careful of feature creep.

Multiple resolutions of the same texture for high/mid/low hardware. Again, this processing can be done by pipeline tools.

Isn't this implicit in the ability to have different materials, each of which can have different parameters, e.g., with different resolution textures?

I don't think so, because mesh primitive can reference only one material. So it would be an application specific approach to switch them.

Yes, exactly. I believe that is standard practice for many apps.

Any LOD/streaming approach should first be proposed as an extension before it would go into core spec, especially a minor spec revision.

In general, LOD approaches are so far and wide (for example, we built 3D Tiles on top of glTF and others are building a progressive approach).

I believe it's kind of a chicken-egg problem. If format doesn't provide enough functionality why should someone use glTF instead of creating a proprietary format?

The ecosystem: exporters, converters, optimizers, etc.

glTF is also extensible so users can add content in extra property objects.

[lexaknyazev]

First draft of texture updates proposal

Base principles

1. Preserve compatibility with existing ecosystem and assets as much as possible.
2. Use existing glTF concepts.
3. Be aligned with WebGL API.

Spec changes for uncompressed textures

Define texture object as such (fields sorted as in texImage2D)

	Type	Description	Required
target	integer	The target that the WebGL texture should be bound to.	No, default: 3553
internalFormat	integer	The texture's internal format.	No, default: 6408
width	integer	The texture's width in pixels.	Depends, see below
height	integer	The texture's height in pixels.	Depends, see below
format	integer	The texture's format.	No, default: 6408
type	integer	Texel datatype.	No, default: 5121
source	string	The ID of the image used by this texture.	Depends, see below
bufferView	string	The ID of the bufferView used by this texture.	Depends, see below
sampler	string	The ID of the sampler used by this texture.	✅ Yes
name	string	The user-defined name of this object.	No
extensions	object	Dictionary object with extension-specific objects.	No
extras	any	Application-specific data.	No

source and bufferView fields are mutually exclusive. Presence of bufferView requires width and height fields. If neither source nor bufferView are supplied and no extension is present, then texture object is invalid.
texture object mirrors polymorphic nature of WebGL texImage2D function, so it can be used in one of two following ways:

1. WebGL only (current one)

   "texture_Image0001": {
      "target": 3553,
      "format": 6408,
      "internalFormat": 6408,
      "type": 5121,
      "sampler": "sampler_0",
      "source": "Image0001"
   }

2. Traditional

   "texture_Image0001": {
      "target": 3553,
      "format": 6408,
      "internalFormat": 6408,
      "width": 256,
      "height": 256,        
      "type": 5121,
      "sampler": "sampler_0",
      "bufferView": "bufferView_image01"
   }

Updates for compressed textures (KHR_compressed_texture early thoughts)

Support of GPU compression texture formats becomes straightforward after merging aforementioned changes. There is no one-size-fits-all compression format, so we can expect assets to have more than one compressed version of each texture. glTF extensions seem to be the right way to provide them no matter what original format was (web or uncompressed). Fields of each extension object are aligned with compressedTexImage2D function.

The following example enables support for two compressed formats and provides backwards-compatible fallback.

{
  "glExtensionsUsed " : ["WEBGL_compressed_texture_s3tc", "WEBGL_compressed_texture_pvrtc"],
  "texture_Image0001": {
    "sampler": "sampler_0",
    "source": "Image0001", // Web format for fallback
    "extensions": {
      "KHR_compressed_texture": {
        "WEBGL_compressed_texture_s3tc": {
          "internalFormat": 33776, //COMPRESSED_RGB_S3TC_DXT1_EXT
          "width": 256,
          "height": 256,
          "bufferView": "bufferView_s3tc01"
        },
        "WEBGL_compressed_texture_pvrtc": {
          "internalFormat": 35840, //COMPRESSED_RGB_PVRTC_4BPPV1_IMG
          "width": 256,
          "height": 256,
          "bufferView": "bufferView_pvrtc01"
        }
      }
    }
  }
}

Proposed extension object schema (internalFormat, width, height and bufferView) fits S3TC, ETC1, PVRTC & ATC compression formats (all have WebGL implementations) and can be used later for WEBGL_compressed_texture_es3 and WEBGL_compressed_texture_astc as well.

Q: Should keys of KHR_compressed_texture dictionary be aligned with extension names? It can make texture selection easier for runtime.

Updates for float textures

5126 (FLOAT) is only allowed for texture.type when glExtensionsUsed contains OES_texture_float.
36193 (HALF_FLOAT_OES) is only allowed for texture.type when glExtensionsUsed contains OES_texture_half_float.

---

@pjcozzi What do you think?

[lexaknyazev]

A few more updates on texture validity:

1. Not all combinations of format and type are valid. The combinations accepted by the GL are defined in OpenGL ES 2.0, table 3.4 (including combinations from OES_texture_float if applicable).
2. bufferView.byteLength must be equal to width * height * bytesPerPixel, where bytesPerPixel is defined in OpenGL ES 2.0, table 3.4 (including combinations from OES_texture_float if applicable) for used format and type combination.
3. KHR_compressed_texture/compressedTexture/bufferView.byteLength, width & height must obey format-specific requirements defined in corresponding extension.
4. bufferView can be explicitly set to null to support uninitialized textures (both compressed and uncompressed).

Thoughts?

[lexaknyazev]

Updates for cube mapping support

With aforementioned changes for uncompressed and compressed texture support adding cubemaps support for such formats is as trivial as allowing 34067 (TEXTURE_CUBE_MAP) for texture.target property and making all six textures available in fixed (-X, +X, -Y, +Y, -Z, +Z) order in the referenced bufferView.

It implies that bufferView.byteLength will be six times bigger than it will be for one texture of the same dimensions and format (it's also true for compressed formats). It also implies that texture.width equals texture.height.

Supporting cubemaps is more tricky for web formats such as JPG or PNG as source field is already defined as string referencing id of image.

Proposal:
A new object type cubeImage containing references to six image objects. Such images must be squares and have same dimensions.
The elements of cubeImages root dictionary can be referenced in texture.source when texture.target is TEXTURE_CUBE_MAP.

Example:

{
  "textures": {
    "texture_0001": {
      "target": 34067, // TEXTURE_CUBE_MAP
      "sampler": "sampler_0",
      "source": "CubeImage0001" // reference to instance of cubeImage
    }
  },
  "cubeImages": {
    "CubeImage0001": {
      "sources": [ // string[6], IDs of image objects, same fixed order (-X, +X, -Y, +Y, -Z, +Z)
        "CubeImage0001_NegX",
        "CubeImage0001_PosX",
        "CubeImage0001_NegY",
        "CubeImage0001_PosY",
        "CubeImage0001_NegZ",
        "CubeImage0001_PosZ"
      ]
    }
  },
  "images": {
    "CubeImage0001_NegX": {
      "uri": "negX.jpg"
    },
    "CubeImage0001_PosX": {
      "uri": "posX.jpg"
    },
    "CubeImage0001_NegY": {
      "uri": "negY.jpg"
    },
    "CubeImage0001_PosY": {
      "uri": "posY.jpg"
    },
    "CubeImage0001_NegZ": {
      "uri": "negZ.jpg"
    },
    "CubeImage0001_PosZ": {
      "uri": "posZ.jpg"
    }
  }
}

What do you think?

[pjcozzi]

@lexaknyazev thanks for this proposal.

Questions

Spec changes for uncompressed textures

I know internalFormat is in the WebGL API, but what is the use case for including it? We had it awhile ago in glTF, but removed it, #195.

source and bufferView fields are mutually exclusive.

OK. What is our reasoning for directly using bufferView, not accessor? Because accessor is really for vertices and could require a bunch of CPU processing, e.g., compacting from stride, etc. We just need an answer here since users may ask.

Updates for compressed textures (KHR_compressed_texture early thoughts)

    "extensions": {
      "KHR_compressed_texture": {
        "WEBGL_compressed_texture_s3tc": {
          "internalFormat": 33776, //COMPRESSED_RGB_S3TC_DXT1_EXT
          "width": 256,
          "height": 256,
          "bufferView": "bufferView_s3tc01"
        },
        "WEBGL_compressed_texture_pvrtc": {
          "internalFormat": 35840, //COMPRESSED_RGB_PVRTC_4BPPV1_IMG
          "width": 256,
          "height": 256,
          "bufferView": "bufferView_pvrtc01"
        }
      }

Why are width and height inside the extensions JSON?

Q: Should keys of KHR_compressed_texture dictionary be aligned with extension names? It can make texture selection easier for runtime.

Yes, I think.

Updates for float textures

Sounds good.

Updates for cube mapping support

Will get back to you about this.

[lexaknyazev]

I know internalFormat is in the WebGL API, but what is the use case for including it? We had it awhile ago in glTF, but removed it, #195.

Looks like it's still in spec and schema.

What is our reasoning for directly using bufferView, not accessor? Because accessor is really for vertices and could require a bunch of CPU processing, e.g., compacting from stride, etc. We just need an answer here since users may ask.

Seems fair to clarify it here, because using accessor will add extra runtime processing with no benefits. texImage2D mentions ArrayBufferView, so we're consistent here.

Why are width and height inside the extensions JSON?

Because of two format-specific limitations:

• S3TC requires width and height to be a multiple of 4
• PVRTC requires width and height to be powers of two

[lexaknyazev]

Addition to Mipmapping Implementation Note:
generateMipmap() will throw INVALID_OPERATION error on compressed formats.

[pjcozzi]

Looks like it's still in spec and schema.

ha. You are right. I didn't remember adding it.

[lexaknyazev]

Why are width and height inside the extensions JSON?

Because of two format-specific limitations:

• S3TC requires width and height to be a multiple of 4
• PVRTC requires width and height to be powers of two

There are couple more reasons to have dimensions inside extension JSON:

• PVRTC implementations may require width==height.
• Since original texture can be provided in the form of source reference, presence of texture.width and texture.height can confuse runtime without compression extension support.

[lexaknyazev]

Q: Should keys of KHR_compressed_texture dictionary be aligned with extension names? It can make texture selection easier for runtime.

Yes, I think.

I've two concerns about meaning of those keys and the extension structure:

Names of texture compression extensions are different in WebGL and OpenGL ES. So ES runtime will have to translate them first to be able to render WebGL assets. On the other hand, making those names clear and consistent (i.e., aligning them with WebGL extension registry) seems to be a good way to achieve assets interoperability across different runtimes.

It may be useful (for some specific rare cases) to provide two textures of the same compression format but with different bitrate (2bpp/4bpp). While it makes sense only for S3TC and PVRTC now, situation may change later with ASTC adoption. Again, this is a highly application specific issue, so I don't think it should be addressed here.

[pjcozzi]

There are couple more reasons to have dimensions inside extension JSON...

OK, sounds good.

On the other hand, making those names clear and consistent (i.e., aligning them with WebGL extension registry) seems to be a good way to achieve assets interoperability across different runtimes.

OK with me.

Labeling as 1.1 to keep 1.0.1 scoped for validation. However, I am happy to keep discussing this now.

[lexaknyazev]

With 1.1 we can do even more here: allow depth for texture and extension object when target is TEXTURE_3D or TEXTURE_2D_ARRAY for WebGL 2 profile.

[donmccurdy]

Can this issue be closed?

[lexaknyazev]

Yeah, this is obsolete.