Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If a preferred codec is filtered out, does it still get assigned a PT? #2938

Closed
henbos opened this issue Feb 13, 2024 · 17 comments
Closed

If a preferred codec is filtered out, does it still get assigned a PT? #2938

henbos opened this issue Feb 13, 2024 · 17 comments

Comments

@henbos
Copy link
Contributor

henbos commented Feb 13, 2024

Use case: I want to signal in the SDP that I support sending X, but my transceiver is sendrecv.

X is filtered out from the preference list at createOffer, so it's not in the m=video line. But should it be in the a=rtpmap list?

Arguments for:

  • It's useful and avoids having to signal things outside SDP.

Argument against:

  • JSEP: "media formats MUST be generated in the corresponding order and MUST exclude any codecs not present in the codec preferences"
@alvestrand
Copy link
Contributor

I think JSEP did not think about this use case.
For subsequent offers, the rule is different:

The "m=" line and corresponding "a=rtpmap" and "a=fmtp" lines MUST only include media formats that have not been excluded by the codec preferences of the associated transceiver and also MUST include all currently available formats. Media formats that were previously offered but are no longer available (e.g., a shared hardware codec) MAY be excluded.

I think JSEP should have an errata suggesting a replacement with "MUST be generated in the corresponding order and MUST include all codecs present in the codec preferences, plus any codecs that need a PT assigned (for example a send only codec)".

@henbos
Copy link
Contributor Author

henbos commented Feb 13, 2024

That errata makes sense to me.

Though we need to have setCodecPreferences not throw for send only codecs for this to work. I thought SCP only rejected if there was no recv codec in the list, but it currently throws if any codec is sendonly even if you have receive codecs in the list too.

Let codecCapabilities be RTCRtpReceiver.getCapabilities(kind).codecs.

For each codec in codecs,

If codec does not match any codec in codecCapabilities, throw InvalidModificationError.

@fippo
Copy link
Contributor

fippo commented Feb 13, 2024

I believe JSEP had a specific intention (but we should really nag the authors).
If your initial offer included a codec that you were capable of sending but not receiving how would you signal that?

The "m=" line and corresponding "a=rtpmap" and "a=fmtp" lines MUST only include ..

That covers the following scenario:
A: H264-only offer
B: H264-only answer (but might include additional codecs, unless sCP was used)
B: H264 + VP8 offer

Do we have tests for that? I doubt it but that is fixable.

@rshpount
Copy link

I think the general intent with the SDP offer/answer is to include codecs that you can only send in the sendonly m= line, codecs that can only receive on the receiveonly m= line, and only list codecs you can both send and receive on the sendrecv m= line. SDP is not designed to negotiate different codecs being used for sending and receiving on the same m= line.

@henbos
Copy link
Contributor Author

henbos commented Feb 14, 2024

If we go with #2939 I think this largely becomes a non-issue and we'd be consistent with @rshpount's interpretation of SDP. E.g. if you want to signal it you restrict transceiver direction and if you don't you can always add it in the answer as a new PT but that's less straightforward use of the APIs.

chromium-wpt-export-bot pushed a commit to web-platform-tests/wpt that referenced this issue Feb 15, 2024
Asserting the "and also MUST include all currently available formats"
https://www.rfc-editor.org/rfc/rfc8829.html#name-subsequent-offers

See w3c/webrtc-pc#2938

BUG=None

Change-Id: I7748ea94e58ba93cdf1043ee0235a7e560e260e2
@cdh4u
Copy link

cdh4u commented Feb 15, 2024

I think the general intent with the SDP offer/answer is to include codecs that you can only send in the sendonly m= line, codecs that can only receive on the receiveonly m= line, and only list codecs you can both send and receive on the sendrecv m= line. SDP is not designed to negotiate different codecs being used for sending and receiving on the same m= line.

Correct.

@aboba
Copy link
Contributor

aboba commented Feb 15, 2024

Yes. It is laid out in RFC 3264 Section 5.1:

"For a sendonly stream, the offer SHOULD indicate those formats the offerer is willing to send for this stream. For a recvonly stream, the offer SHOULD indicate those formats the offerer is willing to receive for this stream. For a sendrecv stream, the offer SHOULD indicate those codecs that the offerer is willing to send and receive with."

@alvestrand
Copy link
Contributor

@rshpount we seem to have current applications that have sendonly codecs that they expect to be able to use for sendrecv lines - in webrtc scenarios, the media streams are treated as unidirectional, but can be paired up to use a sendrecv media section, without that creating any other binding between them - EXCEPT that they now share a set of codecs.

If it was feasible to outlaw sendrecv m-lines under Unified Plan, I'd be happy to do so - but I don't think that's Web compatible at this point.

@rshpount
Copy link

rshpount commented Feb 19, 2024

@alvestrand The right thing would be to get rid of SDP and offer/answer and replace them with an API service that specifies which codecs should be sent and received from each transceiver. This is, obviously, a breaking change.

The issue with the offer/answer is that between the time the offer is generated and the answer is received, the offerer should expect to receive any of the codecs listed in the sendrecv m= line. Once the answer is received, the offerer can discard all the codecs not included in the answer for the sendrecv line, which means using codecs not included in the answer is not allowed in either direction. If the codec is present in the answer sendrecv m= line, the sender can send it. It does not matter that the codec is not the first in the sequence. The sequence of codecs in the m= line is just a preference but not a hard requirement. Also, the answerer can add codecs in the answer that were not present in the offer. The offerer can send media using these codecs as well. Finally, the answerer can send using any of the codecs present in both the offer and answer and switch between them as appropriate. The only way for the offerer to force a smaller set from the original set was to initiate another offer/answer.

So, what this comes down to is that an offer/answer negotiation can set up a list of common codecs (with a small caveat that the answerer can add unidirectional codecs that it can receive). You can add an API surface to force sending using a specific codec from the negotiated list. That API is not a codec preference. It would be a straight codec selection.

@alvestrand
Copy link
Contributor

@rshpount this (getting to an SDP-free future) is part of the motivation behind #2935 - getting to a description of the codec situation that is SDP-independent.

The API surface for setting a specific sending codec is documented in https://w3c.github.io/webrtc-extensions/#dom-rtcrtpencodingparameters-codec - invoked by SetParameters() on the sender, just as one would expect in an SDP-free environment. It does not trigger "negotiation needed".

The question that this issue is raising is the case where, for codec "S", which we only support for sending, how, in an initial offer, do we allocate a PT for that and request that the responder signal its willingness to accept that codec?

This offer will do it (as long as the responder supports S and hasn't set codec preferences):

m=video .... 94 ....
a=sendrecv
a=rtpmap:94:S

But it ALSO tells the receiver that we're willing to receive S, which is not true.

Stating that this is impossible in SDP and we're OK with that is fine with me; there are worse things in the world.
But it should be an explicit decision to say that this is not possible.

@alvestrand
Copy link
Contributor

My current proposal:

Add a NOTE saying:

"NOTE: Using SDP, it is not possible to indicate in a media section with direction "sendrecv" of an offer that you support a codec for sending that you don't support for receiving. This is a limitation of SDP."

@stefhak
Copy link
Contributor

stefhak commented Feb 20, 2024

I agree to your conclusion @alvestrand and like the proposed note. Likely there should be some reference to an SDP RFC as well, but that's just a detail.

chromium-wpt-export-bot pushed a commit to web-platform-tests/wpt that referenced this issue Feb 20, 2024
Asserting the "and also MUST include all currently available formats"
https://www.rfc-editor.org/rfc/rfc8829.html#name-subsequent-offers

See w3c/webrtc-pc#2938

BUG=None

Change-Id: I7748ea94e58ba93cdf1043ee0235a7e560e260e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5293377
Commit-Queue: Philipp Hancke <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1262601}
chromium-wpt-export-bot pushed a commit to web-platform-tests/wpt that referenced this issue Feb 20, 2024
Asserting the "and also MUST include all currently available formats"
https://www.rfc-editor.org/rfc/rfc8829.html#name-subsequent-offers

See w3c/webrtc-pc#2938

BUG=None

Change-Id: I7748ea94e58ba93cdf1043ee0235a7e560e260e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5293377
Commit-Queue: Philipp Hancke <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1262601}
@rshpount
Copy link

I agree with the note. I would expand it a little bit, saying that:
"NOTE: Using SDP, it is not possible to indicate in a media section with the direction "sendrecv" of an offer that you support a codec for sending that you don't support for receiving. This is a limitation of SDP. Such information should be conveyed via a side channel, outside of the SDP offer/answer negotiation."

@dontcallmedom-bot
Copy link

This issue was mentioned in WEBRTCWG-2024-02-20 (Page 18)

@alvestrand
Copy link
Contributor

Should we add to the note ", or sendonly / recvonly media sections can be used"?
There's no problem with signalling the unidirectional codecs if the direction is known.

@cdh4u
Copy link

cdh4u commented Feb 21, 2024

I agree with the note. I would expand it a little bit, saying that: "NOTE: Using SDP, it is not possible to indicate in a media section with the direction "sendrecv" of an offer that you support a codec for sending that you don't support for receiving. This is a limitation of SDP. Such information should be conveyed via a side channel, outside of the SDP offer/answer negotiation."

I don't think we should talk about side channels ,outside of SDP, etc.

I also think it is important to point out that it is not a limitation of SDP - SDP allows you to indicate direction-specific capabilities using sendonly and recvonly. It is a limitation of "sendrecv" m- lines, and the old offer/answer assumption that you only need to indicate what you can receive.

marcoscaceres pushed a commit to web-platform-tests/wpt that referenced this issue Feb 23, 2024
Asserting the "and also MUST include all currently available formats"
https://www.rfc-editor.org/rfc/rfc8829.html#name-subsequent-offers

See w3c/webrtc-pc#2938

BUG=None

Change-Id: I7748ea94e58ba93cdf1043ee0235a7e560e260e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5293377
Commit-Queue: Philipp Hancke <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1262601}
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue Feb 28, 2024
…quent offers, a=testonly

Automatic update from web-platform-tests
webrtc wpt: add test for codecs in subsequent offers

Asserting the "and also MUST include all currently available formats"
https://www.rfc-editor.org/rfc/rfc8829.html#name-subsequent-offers

See w3c/webrtc-pc#2938

BUG=None

Change-Id: I7748ea94e58ba93cdf1043ee0235a7e560e260e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5293377
Commit-Queue: Philipp Hancke <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1262601}

--

wpt-commits: 2898a8f1b5581668cc29ea8fefe3d76f98930048
wpt-pr: 44603
ErichDonGubler pushed a commit to erichdongubler-mozilla/firefox that referenced this issue Feb 28, 2024
…quent offers, a=testonly

Automatic update from web-platform-tests
webrtc wpt: add test for codecs in subsequent offers

Asserting the "and also MUST include all currently available formats"
https://www.rfc-editor.org/rfc/rfc8829.html#name-subsequent-offers

See w3c/webrtc-pc#2938

BUG=None

Change-Id: I7748ea94e58ba93cdf1043ee0235a7e560e260e2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5293377
Commit-Queue: Philipp Hancke <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1262601}

--

wpt-commits: 2898a8f1b5581668cc29ea8fefe3d76f98930048
wpt-pr: 44603
@henbos
Copy link
Contributor Author

henbos commented Oct 2, 2024

At TPAC we decided to keep things simple, see #3006 with links and summary of discussions. Closing this issue

@henbos henbos closed this as completed Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants