Idea: Protocol plugins, client/server capability handshake

[ericyhwang]

These are some of my thoughts around enabling more extensibility of ShareDB and more rapid development of new ideas. I'm hoping to find some time in the next month or two to write a more fleshed out spec, but in the meantime, comments and more ideas are welcome!

ShareDB message protocol plugins

Over the past couple years, there have been new features that require extending the ShareDB message protocol with different request.a "actions":

• fetchSnapshotByTimestamp
• Presence, still in progress

These are handled in large switch statements:

• On the server in Agent: https:/share/sharedb/blob/v1.1.0/lib/agent.js#L334
• On the client in Connection: https:/share/sharedb/blob/v1.1.0/lib/client/connection.js#L183

I've been thinking it would be good to implement a framework for message protocol plugins, and refactor more optional features like queries, fetchSnapshotByTimestamp, and presence into plugins. This has some nice advantages:

1. ShareDB users could choose exactly what features they want, keeping code and bundle sizes smaller.
2. We can develop new protocol-level features more rapidly as independent plugins, compared to developing them in ShareDB core. They can also be independently semver'd.
3. ShareDB users could author their own protocol plugins without needing to touch ShareDB core.

One of the requirements to accomplishing this is that clients and servers need to be able to do some sort of capability handshake, so they can agree on what protocol plugins to use for the session and at what versions.

Client/server capability handshake

I haven't put much thought into this yet. It could work like a simpler version of a TLS handshake. It would need to include at least:

• Capability names
• Capability versions

Potential use cases for a capability handshake:

• ShareDB message protocol plugins
• Making it easier to do major semver updates to ShareDB core
• Negotiating a default OT type or supported OT types in general: A specific version of ot-json0 is hardcoded as the basic type #284
• Custom application-specific capabilities defined by ShareDB users

[curran]

It's a great idea! ShareDB needs this.

[alecgibson]

Vague high-level brain dump:

We'll need to think about what to do in non-happy handshake cases. For example, it may be acceptable to run without some plugins, but not without others. Even here, the server and client may disagree about what's important. In the case where a plugin isn't loaded, but this is deemed acceptable, it would be nice to inform the client, so they can eg alert the user about reduced functionality / prompt them to reload the page, etc. Similarly if an older version of a plugin is agreed upon, the client may want to handle this in some way.

Plugins will need access to the instances of Agent and Backend. We'll need to think about how plugins access the database: what if they need to extend driver functionality? Do we want to restrict what they do in any way? Do plugins need to be "sandboxed" from one another to reduce interference? Presumably there aren't any security concerns, because you've actively installed this plugin yourself?

We may want to add some sort of formalised pubsub channel namespacing? And maybe a consistent format for middleware payloads?

We'll also want to think about how plugins will be accessed. Do we recommend people extend the Connection prototype (feels weird)? Or do something like connection.plugins.presence.getPresence() (which is a bit verbose)? Or do plugins just get their own instance anyway eg const presence = new Presence(); connection.use(presence) (which requires consumers to actively register instances on both client and backend, which is again a bit verbose)?

[ericyhwang]

Rough ideas and notes from discussion today:

• Client reports the features it supports, including versions. If a client supports multiple versions of a feature, it lists the versions in order of preference.
• Server responds with, out of that set, the features that it accepts and the version of each feature that it wants to use. If the server doesn't want to support an old version of a feature, it should have some way of indicating that, so the client can tell the user to upgrade.
• Based on the server response, client application code can decide how critical each feature is. It could prompt the user to upgrade or to continue with missing optional functionality.

OT type negotiation:

• Client and server should agree on a default type for the connection, which the server should store on the agent.
• Versioning OT types will be helpful for things like rolling out presence, where the client needs the server's OT type to support the presence transformations. Leaning towards having a version field exported on each OT type, defaulting to 1.0. Version field could be automatically produced from the package.json?

[alecgibson]

You can keep the OT type version in sync with package.json using a magical variable, although I suspect this would need some build pipeline, instead of just publishing raw JS files. I've also seen some funky things like require('package.json').version, but I suspect that requires Webpack or similar?

module.exports {
  name: 'json0',
  uri: 'http://sharejs.org/types/JSONv0',
  version: process.env.npm_package_version,
}