Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KHR_texture_video #2285

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

KHR_texture_video #2285

wants to merge 5 commits into from

Conversation

dwrodger
Copy link

This is a proposal for an extension to standardize the way that video textures can be assigned in gltf. Currently video textures are supported by most standard 3D players on the web, but the configuration of the texture requires custom javascript unique to each player. This extension intends to standardize that.

Copy link
Contributor

@aaronfranke aaronfranke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am interested in video textures, but the spec is very ambiguous and has errors.

@@ -0,0 +1,155 @@
# KHR_texture_video
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file should be named README.md

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I have fixed it in the latest commit.


## glTF Schema Updates

The `KHR_texture_video` extension is added to the `textures` node and specifies a `source` property that points to the index of the `images` node which in turn points to the video.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

points to the index of the images node which in turn points to the video.

How does images in turn point to a video? In your example, the entirety of the "images" only contains one object with a reference to a PNG file:

"images": [
    {
        "uri": "image.png"
    }
]

I think what you meant is that "source": 0, inside of "KHR_texture_video": { points to /extensions/KHR_texture_video/videos/0, but this is not at all clear from the document.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I have updated this section to be more clear and precise in the latest commit. Please let me know if it is clear and unambiguous now.

Comment on lines 45 to 47
"videos": [
"uri": "texture.webm"
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not valid JSON, the [] symbols specify an array but you are defining a property like a sub-JSON.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I believe it is resolved in the latest commit. Please let me know if you see any further problems with this.


This extension allows glTF models to use videos as textures. A client that does not implement this extension can ignore the provided video and continue to rely on the PNG and JPG textures available in the base specification. Defining a fallback texture is optional.

Video textures support use cases such as water or other textures that continuously repeat, coordination of video texture playback by animation (in conjunction with `KHR_animation_pointer`), or control of video texture playback by interactivity (in conjunction with `KHR_interactivity`).
Copy link
Contributor

@aaronfranke aaronfranke Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is KHR_interactivity defined? It's not a part of this repository.

For interactivity, note that OMI's physics extensions already define an interactible trigger. You can specify a trigger shape that a user can point at in the world and interact with. What actually gets triggered is defined by other extensions, so this video player extension could define that a given OMI trigger plays the video. I would highly recommend that we move forward with OMI physics so that we can use these extensions together (we could rename to have the KHR_ prefix if Khronos is in favor).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KHR_interactivity is an ongoing work by the 3d format's group's DTSG on behaviors and interactivity. It is still an early work in progress, but a partial draft should be available on a branch soon.

Comment on lines 47 to 49
"videos": [
"uri": "texture.webm"
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still not valid JSON.

Copy link
Author

@dwrodger dwrodger Apr 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. Thanks, I believe it's fixed in the latest commit.

@ibouazizi
Copy link
Contributor

There is an MPEG extension MPEG_texture_video, which serves the same purpose but defines the details of how the video format looks like and how it is fed dynamically. Please consider using that extension.

@dwrodger
Copy link
Author

Regarding the vendor extensions MPEG_texture_video and related extensions, it looks like there were two parallel efforts on video textures around the same time. I could see a potential consolidation where we make one extension that defines how video textures are referenced and time-controlled (e.g. linkage from texture and autoplay / connection to KHR_animation_pointer) and potentially another set of extensions that add to that support for specific formats and codecs and provide the rest of the details that this extension provides.

I would like to make sure that the video texture control is designed to work well with KHR_animation_pointer (and the upcoming interactivity work that ties heavily with that). If some example jsons were added to the MPEG_texture_video set of specs it might make it easier to understand whether it can already work well with KHR_animation_pointer or, if not, how it could be adapted to do so.

@gurdep
Copy link

gurdep commented May 10, 2023

@dwrodger the following is an example of MPEG_texture_video which might help for better understanding. The format description, codecs, mime-type, tracks, and other information can be exposed via MPEG_media extension as shown in the example.

@aaronfranke
Copy link
Contributor

@gurdep Those JSON files you linked are just small sections, do you have any links to complete example glTF files?

@gurdep
Copy link

gurdep commented May 11, 2023

@gurdep Those JSON files you linked are just small sections, do you have any links to complete example glTF files?

You may ask the MPEG group for complete glTF files at their official github repository.

@kzhsw
Copy link

kzhsw commented Aug 25, 2023

Since many video container format, like webm in given example, could also include audio tracks, should the behavior of audio tracks in video files be standardized here?

Behaviors might include some of these:

  • default audio track
  • muted or not
  • volume
  • should video autoplay be blocked by browser for having audios, or should it fallback to muted autoplay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants