EXT_cubemap_textures and EXT_mipmap_textures

[MiiBond]

I'm currently sketching out some extensions that add the ability to reference cubemap images and also custom mipmaps from within glTF and I'd like some feedback. For context, we (Adobe) are interested in using this in conjunction with the KHR_environments extension being discussed #946. IBL environments will require referencing cubemaps with custom mip chains (for storing the pre-filtered IBL data).

"textures": [  
  {  
    "extensions": {  
      "EXT_cubemap_textures": {  
        "pos_x": 1,  
        "neg_x": 2,  
        "pos_y": 3,  
        "neg_y": 4,  
        "pos_z": 5,  
        "neg_z": 6,  
      }  
    }  
  } , 
  { 
    "source": 0, 
    "extensions": {  
      "EXT_mipmap_textures": { 
        "mip_sources": [ 1, 2, 3, 4, …] 
      } 
    }  
  },  
...
]  

Note that the cubemap extension references textures (not images) for the 6 faces so that those faces can include separate mip chains. Is this weird? It also allows a particular sampler to be specified for each face though I can't think of a reason that anyone would want that...

What about DDS and KTX?

It's possible to store all of this (6 faces and custom mips) in a single DDS file using the existing MSFT extension (and a future KTX extension?) so these extensions should also be designed to work in tandem with that. Not sure how to do this though...

If the mipmap extension was part of the image and the cubemap faces were image references:

"textures": [  
  {  
    "extensions": {  
      "EXT_cubemap_textures": {  
        "source_pos_x": 1,  
        "source_neg_x": 2,  
        "source_pos_y": 3,  
        "source_neg_y": 4,  
        "source_pos_z": 5,  
        "source_neg_z": 6,  
        "extensions": {  
          "MSFT_dds_textures": {  
            "source": 0 
          } 
        } 
      }  
    }  
  }  
], 

"images": [ 
  { 
    "uri": "my_cube_with_mips.dds", 
    "extensions": {  
      "EXT_mipmap_textures": { 
        "mip_uris": ["my_mip_1.jpg", "my_mip_2.jpg", ...], // or "mip_buffers": […] for GLB 
      } 
    }  
  },  
... 
]  

Yikes.

The runtime would get the DDS, if the MSFT extension is supported. Otherwise, it would grab the individual faces. For the mipmaps, if the "uri" points to a DDS, maybe the runtime would download the DDS first and, if it doesn't include mips, only then would the mipmap extension be looked at for that image.

Or maybe we forget all this and just rely on MSFT_dds_texture as being the only way to support cubemaps or custom mip chains?

Thoughts?

[pjcozzi]

@MiiBond thanks for kicking this all. Also, see #835. Perhaps move that related discussion here.

[lexaknyazev]

@MiiBond In addition to #835, I'd recommend to read discussions in #739 that cover most questions re: KTX/DDS features.

[msfeldstein]

It feels like we're trying to solve one problem (IBL) with three loosely connected extensions. Should this all be rolled up into environments or are they each compelling extensions separately?

[msfeldstein]

Does mip_sources reference images? Why not have EXT_cubemap_textures reference an array of images, to encompass all the mips directly in it? I don't see a use case for EXT_mipmap_textures outside of this PMREM IBL case.

[inigoquilez]

On "Note that the cubemap extension references textures". I think that's not the best option. I think the cubemap should reference Images (not textures) - 6*N of them, with N being the number of mip levels per cubemap. Conceptually, and hence at OpenGL level as well, a cubemap is a texture on its own, with all faces and mips included. It is not a collection of 6 textures with potentially disparate number of mip levels or filtering options (which would not even be implementable in WebGL or any other API and Hardware)

I am not familiar with the limitations of the current design or the backward compatibility requirements of the format. But assuming there are no such constrains, I'd not go with the suggested texture referencing, but image referencing.

[MiiBond]

It feels like we're trying to solve one problem (IBL) with three loosely connected extensions. Should this all be rolled up into environments or are they each compelling extensions separately?

That's possible, of course, @msfeldstein. The intent in separating them was to allow for other use cases.

Being able to separately define custom mips has a variety of uses. I've done this in the past for effects where you don't want lower mips to be filtered with either bilinear or nearest. For example, if you have thin lines that you want to stay the same pixel-thickness as you go down the mip chain. In these cases, an artist might create their own mip chain and want to import that into an engine.

HDR textures (from #1365) could be used for emissive textures in conjunction with some future extension for, say, light maps, height fields or something else.

Cube maps are usually used for some form of environment mapping but I don't think we'd want to roll this into a generic environments extension. Skyboxes, for example, could use cubemaps and don't require custom mips or HDR (necessarily).

I'd not go with the suggested texture referencing, but image referencing.

Agreed, @inigoquilez. I actually suggested image-referencing in my second example. The idea of referencing textures came about because I was thinking of making KHR_hdr_textures (#1365) be declared on the texture to match the way MSFT_dds_textures works. In order to do that, the cubemap would need to be composed of other textures. However, I ended up not going that route so I probably should have just removed the first example. By referencing images, we lose the ability to reference DDS images as individual cube faces but that's not really that important as a single DDS file can encapsulate an entire cubemap.

[lexaknyazev]

Closing, since the work on supporting additional texture features and formats is happening in other threads.