Advanced Codec Capabilities API

[henbos]

Use case:
Allow applications to make smarter decisions when choosing codecs taking implementation capabilities into account, such as preferring hardware accelerated implementations over non-hardware accelerated ones.

Proposal:
New API that allows enumerating encoder and decoder implementations, not just codecs. Each implementation could say if it is HW accelerated or not and what its hard limits are in terms of resolution etc. Ideally this might also say something about how "performant" one can expect the implementation to be, but just knowing HW is useful. This could allow applications to...

• Better support low-end devices.
• Improve battery life by utilizing hardware.
• Discover limitations (e.g. min and max resolution) of particular implementations prior to negotiating.
• Avoid "bad" implementations if the implementation can identify itself. Maybe an implementation X is good for conferencing use cases but has too high latency for gaming use cases.

Alternatives:
WebRTC 1.0 has RTCRtpCodecCapabilities containing RTCRtpCodecCapability. This gives us mimeType and fmptLine, i.e.

• Codec
• Codec parameters

But it could be extended, e.g. "isHardwareAccelerated"

[henbos]

Feedback from TPAC-9-19-2019: Define use cases in a PR

[henbos]

I would assign myself and add labels but I don't have those repo permissions it seems :) @aboba @dontcallmedom

[henbos]

Note to self: Also include Chris Cunningham of Media Capabilities API into the discussions, this is very related to that API.

[dontcallmedom]

done @henbos

[henbos]

I still can't assign myself the issue :'(

[aboba]

@henbos Has this issue been addressed by the Media Capabilities API?

[henbos]

I think the combination for Media Capabilities (giving us an expectations of HW capabilities per codec) and getStats() telling us whether or not we are, in fact, getting HW (powerEfficientEncoder/Decoder) should be enough to close this issue.

[henbos]

Re-open: I guess it should still be listed as a use case?

[aboba]

@henbos Once we define the requirement, we can figure out if where it belongs (e.g. in a new use case, or in existing ones). But I do think it would be useful to have a requirement, if only to understand whether MC and getStats() is sufficient to meet it. Personally, I'm not sure because we haven't been able to figure out how to replace getCapabiltiies() with MC in WebRTC-SVC.

[aboba]

Closing. Please re-open if there is something missing.