Shard master process.

[erikjohnston]

Currently we have a single master process which can quickly become a bottleneck, so we want to be able to run multiple "master" nodes and scale horizontally.

Some notes:

• We have a concept of "streams" that are a sequence of items assigned a monotonically increasing ID, where there is a "max position" which guarentees that every item with an ID below that has been successfully persisted (even though some items with a higher ID may have been persisted).
• Replication between nodes is currently a fairly noddy TCP protocol where all nodes connect to the master process and get streamed updates. This approach will not work in a world where we have multiple "master" nodes.

Potential approaches:

• Move as much as possible onto workers, but retain a single "master" process that does nothing but orchestrate e.g. tracking max pos of streams, streaming updates to workers
• Use an external process, e.g. redis, rabbitmq, etc to manage these things for us.

[djschilling]

If there are steps in this direction please also consider not only performance, but high availability.

So a single master process would not be ideal in my eyes.

See also this issue i opened with focus on ha: #7076

[anoadragon453]

@djschilling The point of this project is to run more than one master process.

That said, ensuring the whole thing doesn't fall over if 2/3 aren't running would be great.

[erikjohnston]

Synapse now supports moving writing of events to a different process. I've opened #7593 to track further work.