Draft: multiselect 2.0 design doc

[marten-seemann]

This document is not the multistream 2.0 protocol spec. It is a design doc to serve as a prelude to a subsequent spec.

We aim to lay out the problem domain first by providing context, pointing out issues, and setting the requirements and possible design directions.

Given the complexity and criticality of this component, it is indispensable to lock consensus about the problems and challenges to solve, before jumping into addressing them via a concrete implementation spec.

[vyzo]

I want to stress simultaneous open support as a critical component.

[bigs]

@raulk @marten-seemann So, I've taken some time to read through this and to revisit @Stebalien's multiselect 2.0 and muse on how that might work with message orientation work. I've come to the conclusion that it should be quite successful, though in order to fulfill the message-oriented use case, there must exist an extension or superset protocol that supports multiplexing. My reasoning is that, in order to have as compact a protocol as possible (to respect MTUs, etc), we would like to have a multiplexer that is aware of the dynamic protocol to ID mappings established through multiselect. If we were to extract the multiplexer into a separate protocol that was negotiated with multiselect, it would have to have access to multiselect's internal mappings, which smells to me.

If you're all interested and with me on this, I'd like to go ahead and add an extension to the multiselect protocol (that streaming transports could ignore) that supports transmission of messages on negotiated protocols. Some initial thoughts:

• The most basic implementation of this could include a single extension to multiselect 2.0 as it stands now, multiselect/message (or something like that), which contains a payload tagged with a protocol ID as negotiated by multistream/dynamic.
• We could evolve (or add a message-oriented copy of) multistream/dynamic to not rely on a previous advertise, instead allowing for the dynamic ID to be established in the dynamic message, e.g. <multistream/multicodec><multistream/dynamic><varint delimited protocol name><varint dynamic ID> and to optionally include a speculative payload.

[yusefnapora]

Hey, just wanted to type up some thoughts after a good chat with @marten-seemann on Friday.

We thought it would be a good idea to kick off some discussion around the tentative points in the doc before we move on to something more concrete

Here's the points in question:

Multiplexer negotiation without "early data"

Sending our supported multiplexers in the early data of TLS 1.3 or Noise handshake is awesome; it saves a round trip and seems like the best way to take advantage of early data without revealing any sensitive info.

The question is what to do if early data is unsupported by the crypto channel. The doc tentatively proposes an "embryonic stream" which frames an initial stream request alongside a set of supported muxers. The alternative would be to do a full roundtrip negotiation for the multiplexer after the crypto handshake completes and before we select an application protocol.

Marten raised a good point, which is that users that are optimizing for round trip cost can just choose TLS 1.3 or Noise, so it may not be worth optimizing for the case where early data is unsupported.

Speculative "one-of" / XOR selection of protocols

Tentative: When (a) multiple protocols are supported for a given exchange (e.g. in the case of versioned protocols, where a peer supports both old and the new versions), and (b) the peer has no knowledge of what the other party supports (i.e. before either receiving a list of supported protocols, or before trying out both protocols), the peer should be able to eagerly send the request for both protocols within a single message, declaring a precedence, allowing the client to select, keep and respond to at most one.

This one definitely is useful for reducing round-trip cost, especially during protocol upgrades. But it also adds a lot of complexity - we'd essentially have to define a sort of "bootstrapping multiplexer" that can frame multiple stream requests and their initial payload, and define the logic to choose between them.

Is this only useful in the protocol upgrade scenario, or are there other use cases that could benefit?

I should note that a few weeks back I tried writing up some protobufs that could cover the requirements in this doc, which basically defines the kind of bootstrap muxer we'd need. It supports the speculative scenario by having the initiator send StreamHeader messages that include protocol ids, an optional initial data payload, and an optional list of alternative StreamHeaders. If the receiver doesn't support the chosen protocol, it can loop through the alternatives and hopefully find one.

But the way it works is complicated and kind of confusing. Since no muxer has been agreed upon when the initiator sends their request, the responder is the one that actually opens the streams using the muxer (after choosing a supported one from the initiator's set). That's why there's an is_response flag in the StreamHeader message, so that the initiator can correlate the incoming stream with a request sent earlier, instead of treating it as a brand new stream from the other party (a correlation UUID is probably better than a flag, tbh).

This obviously requires keeping some state around and some fiddly logic to make our primordial multiplexer line up with the streams opened by the real multiplexer later. This seems like it might be overcomplicating things, but if we really need the speculative "one-of" protocol selection, then we probably need something like this to make it work.

Other stuff

As a side note, both Marten and I seem to like using protobufs for this sort of thing, since it doesn't add a ton of weight to the binary vs positional fields, and it's a lot easier to extend in the future.

Also, we both like the idea of building up the mapping of string protocol ids to integer codes lazily, meaning that the initial selection of a protocol would always use strings (maybe with an exception for core libp2p protos) and when you ACK the selection you can send a short code that can be used to reselect. So peers that care about saving those bytes over a long-lived connection can build up a mapping table, but it's not required.

Anyway, would love to hear thoughts from everyone. @raulk, I know that you are strongly in favor of the speculative one-of selection process - could you elaborate a bit on the motivation? I think we can definitely make it work if we have to, but maybe we don't have to :)