MSC1767: Extensible event types & fallback in Matrix (v2)

proposals/1767-extensible-events.md

@@ -0,0 +1,347 @@
+# MSC1767: Extensible events in Matrix
+
+##  Problem
+
+ 1. There is no formal mechanism of extending events with additional structured metadata.
+ 2. New events tend to reinvent the wheel rather than being able to reuse existing types.
+ 3. Clients don’t know how to render unknown event types.
+
+This is seriously hindering uptake of new event types in Matrix - whether that’s
+richer data types (stickers, location, calendaring, etc) or IOT-style use cases.
+It also means we are currently using an underspecified solution for rich
+messaging, as there has never been a previous agreed way of a jury rig solution
+for rich messaging as we never previously agreed a way of expressing alternate
+formats of displaying the same event.
+
+## Solution
+
+We group the keys of an event's contents together into different types. This lets
+us address the problem by:
+
+ 1. Making it easy to extend events by adding new types into contents alongside existing ones.
+ 2. Making it easy to reuse existing types by adding new ones alongside existing ones.
+ 3. Letting clients render unknown event types by falling back to types in the event that they *do* recognise.
+
+Events still have a primary type, which is used when sending them in Matrix.
+
+The names of the keys in contents are now expected to be namespaced, given they
+typically refer to types of events. However, for compatibility, we can keep using
+some of the original fields too.
+
+Certain types (e.g. `m.message`, `m.caption`, `m.thumbnail`) which can contain
+multiple alternative payloads in different formats express the alternatives as
+an ordered list; most preferred first.
+
+We provide short-form types (`m.text` and `m.html`) for the most common scenarios
+of an event requiring a plain-text or HTML-formatted representation.
+
+For convenience and compatibility, we keep "body" and "formatted_body" as
+shorthand for plain-text and html-formatted fallback for events.
+
+The various types in an event's contents MUST refer to the same event. Multiple
+events should be linked using a `m.relates_to` reference rather than multiplexing
+into a single event.
+
+Here are proposals for different types of events:
+
+### m.text and m.html
+
+For compactness for this super-common case, we define a short form which is
+equivalent to the longer form `m.message` example below:
+
+```json
+{
+ "type": "m.message",
+ "content": {
+ "m.text": "i am a *fish*",
+ "m.html": "i am a <b>fish</b>"
+ }
+}
+```
+
+### m.message
+
+m.message describes a simple textual instant message.
+
+```json
+{
+ "type": "m.message",
+ "content": {
+ "m.message": [
+ {
+ "mimetype": "text/html",
+ "body": "i am a <b>fish</b>"
+ },
+ {
+ "mimetype": "text/plain",
+ "body": "i am a *fish*"
+ }
+ ],
+ }
+}
+```
+
+### m.file
+
+```json
+{
+ "type": "m.file",
+ "content": {
+ "m.text": "foo.dat (12KB)",
+ "m.file": {
+ "url": "mxc://matrix.org/asdjkhcsd",
+ "name": "foo.dat",
+ "mimetype": "application/octet-stream",
+ "size": 12345
+ }
+ }
+}
+```
+
+### m.image
+
+```json
+{
+ "type": "m.image",
+ "content": {
+ "m.text": "matrix logo (640x480, 12KB)",
+ "m.image": {
+ "width": 640,
+ "height": 480,
+ },
+ "m.file": {
+ "url": "mxc://matrix.org/asdjkhcsd",
+ "mimetype": "image/jpeg",
+ "name": "logo.jpg",
+ "size": 12345
+ },
+ "m.caption": [
+ {
+ "body": "matrix logo"
+ }
+ ],
+ "m.thumbnail": [
+ {
+ "url": "mxc://matrix.org/thumb",
+ "mimetype": "image/jpeg",
+ "width": 160,
+ "height": 120,
+ "size": 123
+ }
+ ],
+ }
+}
+```
+
+XXX: do we need to worry about the schema overlap between m.file/m.image and m.thumbnail?
+do we need to specify how specific thumbnails can have specific captions (or is that effectively displayhints?)
+
+### m.message containing original source text, to allow future edits:
+
+```json
+{
+ "type": "m.message",
+ "content": {
+ "m.message": [
+ {
+ "mimetype": "text/html",
+ "body": "I am a <a href=\"http://www.reddwarf.co.uk\">fish</a>",
+ },
+ {
+ "mimetype": "text/markdown",
+ "original": true,
+ "body": "I am a [fish](http://www.reddwarf.co.uk)",
+ },
+ {
+ "mimetype": "text/plain",
+ "body": "I am a fish < http://www.reddwarf.co.uk/ >",
+ },
+ ]
+ }
+}
+```
+
+### m.message interationalised
+
+```json
+{
+ "type": "m.message",
+ "content": {
+ "m.message": [
+ {
+ "body": "Je suis un poisson",
+ "lang": "fr",
+ },
+ {
+ "body": "I am a fish",
+ "lang": "en_EN",
+ }
+ ]
+ }
+}
+```
+
+The french text is preferred and comes next, and a non-i18n aware client would
+use it. A smarter i18n-aware client would realise the user can't speak french
+and fall through to the next one.
+
+### m.video
+
+```json
+{
+ "type": "m.video",
+ "content": {
+ "m.text": "animated matrix logo (logo.mp4, 1280x720, 1m30s, 23.6MB)",
+ "m.video": {
+ "width": 1280,
+ "height": 720,
+ "duration": 90000,
+ },
+ "m.file": {
+ "url": "mxc://matrix.org/v1d30",
+ "mimetype": "video/mp4",
+ "name": "logo.mp4",
+ "size": 23654321
+ },
+ "m.caption": [
+ {
+ "body": "animated matrix logo"
+ }
+ ],
+ "m.thumbnail": [
+ {
+ "url": "mxc://matrix.org/videothumb",
+ "mimetype": "video/mp4",
+ "animated": true,
+ "width": 128,
+ "height": 72,
+ "size": 12300
+ },
+ {
+ "url": "mxc://matrix.org/thumb",
+ "mimetype": "image/jpeg",
+ "width": 128,
+ "height": 72,
+ "size": 123
+ }
+ ],
+ }
+}
+```
+
+### IOT events (e.g. net.arasphere.temperature)
+
+with text fallback:
+
+```json
+{
+ "type": "net.arasphere.temperature",
+ "content": {
+ "m.text": "The temperature is 37C",
+ "m.html": "The temperature is <font color='#f00'>37C</font>",
+ "net.arasphere.temperature": {
+ "temperature": 37.0
+ }
+ }
+}
+```
+
+### m.calendar.request
+
+This is a deliberately chunky event designed to show a geotagged calendar request.
+
+(In practice we should probably use the IETF jsCalendar format rather than inventing a
+new one here, and to ensure interop with the JMAP folks.
+https://tools.ietf.org/html/draft-ietf-calext-jscalendar-11)
+
+```json
+{
+ "type": "m.calendar.request",
+ "content": {
+ "m.text": "Invite to Party at Bob's House on Feb 7th, 3pm",
+ "m.html": "Invite to Party at <i>Bob's house</i> on Feb 7th, 3pm",
+ "m.calendar.request": {
+ "version": "2.0",
+ "summary": "Party at Bob's house",
+ "vEvent": {
+ "dtStart": {
+ "date": "20180207T150000",
+ "tzId": "Europe/London"
+ },
+ "dtEnd": {
+ "date": "20180207T180000",
+ "tzId": "Europe/London"
+ }
+ }
+ },
+ "m.file": {
+ "uri": "mxc://matrix.org/auiheiuh1eqwd",
+ "mimetype": "text/calendar",
+ "name": "bobs-party.ics",
+ "size": 1246
+ },
+ "m.location": {
+ "uri": "geo:37.786971,-122.399677;u=35",
+ "description": "123 Fake Street"
+ },
+ "m.thumbnail": [
+ {
+ "uri": "mxc://localhost/JWEIFJgwEIhweiWJE",
+ "width": 256,
+ "height": 256,
+ "mimetype": "image/jpeg",
+ "size": 1024000
+ }
+ ],
+ }
+}
+```
+
+This would be nicer with displayhints support, but still works pretty well.
+
+
+## Tradeoffs
+
+It's a bit ugly to not know whether a given key will take a string, hash or array.
+
+It's a bit arbitrary as to which fields are allowed lists of fallbacks.
+
+It's a bit ugly that you have to look over the keys of contents to see what types
+are present, but better than duplicating this into an explicit `types` list (on balance).
+
+We're skipping over defining rules for which fallback combinations to display
+(i.e. "display hints") for now; these can be added in a future MSC if needed.
+MSC1225 contains a proposal for this.
+
+## Issues
+
+I think Erik had some concerns about mixing together types at the top level of `contents`
+but I've forgotten the details. Hopefully these are mitigated by the revised approach.
+
+## Security considerations
+
+We can't apply ACLs serverside to the types embedded in the event contents.
+However, this is inevitable given the existence of E2E, so we have no choice but
+for clients to apply ACLs clientside (e.g. refuse to render an m.image contents
+on an event if the sender doesn't have enough PL to send an m.image event).
+
+## Availability
+
+Given the disruption of migrating clients & bridges to the new event shape, this
+should probably land after S2S r0.
+
+## Migration
+
+Currently visible events are `m.room.message` with a `msgtype` to distinguish
+between `m.text`, `m.image`, `m.file`, etc.
+
+...?
+
+## Changes from MSC1225
+
+ * converted from googledoc to MD, and to be a single PR rather than split PR/Issue.
+ * simplifies it by removing displayhints (for now)
+ * removes all references to mixins, as the term was scaring people and making it feel far too type-theoretic
+ * replaces the clunky m.text.1 idea with lists for types which support fallbacks
+ * removes the concept of optional compact form for m.text by instead having m.text always in compact form
+ * tries to accomodate most of the feedback on GH and Google Docs from MSC1225.