What is the interaction of media capabilities with WebCodecs

[youennf]

https://www.w3.org/TR/media-capabilities/#dom-mediaencodingtype-webrtc states that:
webrtc is used to represent a configuration that is meant to be transmitted using [RTCPeerConnection](https://www.w3.org/TR/webrtc/#dom-rtcpeerconnection) as defined in [[webrtc]](https://www.w3.org/TR/media-capabilities/#biblio-webrtc))

It does not seem that webrtc encoding mode should be used for WebCodecs+WebTransport.
Is there a need to clarify this mode can also be used for other pipelines like WebCodecs+WebTransport?
Or is there some benefit to introduce a specific WebCodecs mode?

[aboba]

AFAICT, there is no interaction. WebCodecs has the isConfigSupported API, which returns whether the configuration is supported, as well as the configuration set by the Audio/Video Encoder/Decoder (which may be a subset of the config passed as an argument). So isConfigSupported provides the information that Media Capabilities would return (other than smooth) and adds information that MC would not provide (the set configuration).

[youennf]

isConfigSupported provides the information that Media Capabilities would return (other than smooth)

And powerEfficient.

[youennf]

Discussed at TPAC, we might want to reword the 'webrtc' category, or mention that it can be also used for live transmission via other APIs like WebCodecs.

[Djuffin]

We can amend the spec to state that this entry also works for RTC use-cases using WebCodecs.
Adding new entry for "webcodecs" seems problematic because WebCodecs gives a lot of control to the website and it's not clear what it would mean in this case.

[aboba]

@Djuffin The answers Media Capabilities provides for 'webrtc' won't necessarily apply to WebCodecs. For example, WebRTC supports spatial scalability modes, but WebCodecs does not.

[aboba]

@youennf isConfigSupported takes the configuration as an argument. So if you set hwAcceleration = "prefer-hardware" and the configuration was supported, wouldn't that indicate that the configuration was powerEfficient?

[Djuffin]

For example, WebRTC supports spatial scalability modes, but WebCodecs does not.

isConfigSupported is the way to figure out if anything is supported.
Media capabilities can be used to check if it's going to be smooth for RTC scenarios.

[aboba]

If Media Capabilities indicates that a config will be smooth for WebRTC, does that necessarily imply that it will be smooth for WebCodecs?

[chrisn]

What would "smooth" mean in relation to WebCodecs, where the rendering of the content is controlled by the application?

[Djuffin]

The way I meant "smooth" for WebCodecs (here #202 (comment)) was

1. sending side (VideoEncoder) can encode a video stream at a given fps
2. receiving side (VideoDecoder) can decode it at a given fps and draw it to 2d canvas

Maybe we should avoid dragging webcodecs into media capabilities.
(Although web developers most likely will use webrtc as a proxy for webcodecs performance anyway)

[Djuffin]

We could also amend VideoEncoder.isConfigSupported and VideoDecoder.isConfigSupported to clarify that if latencyMode=realtime or optimizeForLatency=true it should only return true if the device is capable of encoding/decoding in real time at a given FPS, the only difficulty here is that UA might not have a practical and reliable way to know this (but the same is true for media capabilities as well).

[aboba]

isConfigSupported provides additional functionality beyond what MediaCapabilities provides (config). Since WebCodecs decoder can be used with multiple rendering mechanisms (e.g. canvas, mediacapture-transform, WebGPU), the definition of "smooth" is not as clear as it is with a higher-level API like WebRTC.

[youennf]

Discussed today (minutes) and it seems we could do the following:

• In the short term, mention in the spec the relationship between media capabilities type (webrtc, file...) and webcodec parameters (latencyMode...).
• In the long term, investigate using media capabilities for web codecs (potentially with a WebCodecs config instead of mime type).

[aboba]

Also, we could:

• Clarify the behavior of isConfigSupported. For example, with respect to framerate in RTC scenarios, don't return supported if the encode/decode config indicates a desire for realtime latency but the desired framerate is unattainable.

[youennf]

• For example, with respect to framerate in RTC scenarios, don't return supported if the encode/decode config indicates a desire for realtime latency but the desired framerate is unattainable.

In that case, I would think MC would return not supported for webrtc as well.
In case MC returns smooth=false, I would think it means that the webcodecs corresponding encoder is likely to drop frames but not systematically.

[chrisn]

Minutes from yesterday's (9 Jan 2024) discussion: https://www.w3.org/2024/01/09-mediawg-minutes.html