Overseer: subsystems communicate directly #2227

rphmeier · 2021-01-08T00:10:06Z

The previous behavior was that messages would go first to the overseer and then be relayed onwards to the destination subsystem.

Now, subsystems communicate via channels directly to other subsystems. Additionally, they have the option of either sending a message through a bounded or unbounded channel. The unbounded channels should be used sparingly in situations where:

There is a communication cycle between subsystems
One of the parts of the cycle has a clear bound on the number of messages produced.

A few examples of places we might use send_unbounded_message are when we send locally generated backing statements to statement distribution or when we send assignments and approvals from approval voting to the approval distribution subsystem. This avoids deadlock when a network subsystem is .awaiting a send to the core subsystem, while the core subsystem sends to the networking subsystem at the same time. If buffers are full, that will deadlock. Unbounded channels avoids this situation.

SubsystemContext and the overseer itself have an invariant that when a subsystem sends a message after receiving a signal from the overseer, that the recipient of the message will not receive it until having observed the same signal. This allows subsystems to assume that other subsystems are aware of the same view of blocks as they are.

My change-set here upholds the same invariant by having each subsystem tag each message with the number of signals that it had observed before sending the message. If the recipient has not received at least that many signals, it pockets the message until it has received enough signals. This rests on the fact that the overseer sends signals to all subsystems equally. If one subsystem receives a signal, others can expect to receive it shortly.

I have also given SubsystemContext a SubsystemSender assoc. type which will be used to split sending and receiving. Follow-up PRs will make use of this feature.

eskimor · 2021-01-08T11:24:56Z

Our use of channels is still likely to lead to deadlocks, because we .await on both sides unconditionally. Future refactoring should alter subsystems to time-out messages to other subsystems so that progress can be made even under heavy load.

I think we could do better than this, but I need to check a few things first and think a bit more about it.

coriolinus

Core logic in OverseerSubsystemContext::recv LGTM. I agree that this maintains the message invariant.

I'm pleasantly surprised that making this change didn't require changing any of the individual subsystems.

node/overseer/src/lib.rs

coriolinus · 2021-01-08T11:07:05Z

node/overseer/src/lib.rs

+ signals_received: usize,
+ message: AllMessages,
+ ) {
+ fn make_packet<T>(timer: MaybeTimer, signals_received: usize, message: T) -> MessagePacket<T> {


It's too bad we don't have HKTs yet; if we did, you could simplify this into

let make_packet = for<T> |message: T| { ... }

and simply capture the values of timer and signals_received, which would simplify usage.

Well we can do without, if we are sure this is the way to go, we can just expose the subsystem senders directly.

I had the same thought when I was writing the code, but unfortunately no HKT yet

coriolinus · 2021-01-08T11:12:13Z

node/overseer/src/lib.rs

+ if res.is_err() {
+ tracing::debug!(
+ target: LOG_TARGET,
+ "Failed to send a message to another subsystem",


I suspect that at some point we're going to want to debug which particular subsystem was the intended recipient of the failed message. Maybe an api like

impl AllMessages { pub fn recipient_name(&self) -> &'static str { ... } }

would make that simple to add to this debug message.

eskimor

Looks sensible to me.

drahnr

Implementation wise, it looks good.

As a side effect here we lose the overseer as our probe point for messages though.

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

1. we don't provide good names 2. these names are never used anywhere

We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.

node/overseer/src/lib.rs

node/subsystem/src/lib.rs

ordian · 2021-03-28T09:47:26Z

node/overseer/src/lib.rs

 prometheus::Opts::new(
- "parachain_to_overseer_sent",
- "Number of elements sent by subsystems to overseer",
+ "parachain_overseer_signals_sent",


naming nit: what's the difference between parachain_subsystem and parachain_overseer prefix? They both are parameterized by subsystem_name.

subsystem_signals_sent sounds like the subsystems are sending the signals but they are actually receiving them. overseer_unbounded_sent has the same issue in the opposite direction.

These are OverseerSignals so overseer_signals_sent makes sense I think. But I could rename subsystem_bounded_sent etc.. not sure to what

node/overseer/src/lib.rs

node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

rphmeier · 2021-03-28T15:29:18Z

bot merge

ghost · 2021-03-28T15:29:20Z

Waiting for commit status.

* master: Call NetworkService::add_known_address before sending a request (#2726) Overseer: subsystems communicate directly (#2227) Request based PoV distribution (#2640)

* overseer: pass messages directly between subsystems * test that message is held on to * Update node/overseer/src/lib.rs Co-authored-by: Peter Goodspeed-Niklaus <[email protected]> * give every subsystem an unbounded sender too * remove metered_channel::name 1. we don't provide good names 2. these names are never used anywhere * unused mut * remove unnecessary &mut * subsystem unbounded_send * remove unused MaybeTimer We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly. * remove comment * split up senders and receivers * update metrics * fix tests * fix test subsystem context * fix flaky test * fix docs * doc * use select_biased to favor signals * Update node/subsystem/src/lib.rs Co-authored-by: Andronik Ordian <[email protected]> Co-authored-by: Peter Goodspeed-Niklaus <[email protected]> Co-authored-by: Andronik Ordian <[email protected]>

rphmeier added 2 commits January 7, 2021 18:36

overseer: pass messages directly between subsystems

227e71a

test that message is held on to

aef7f24

coriolinus approved these changes Jan 8, 2021

View reviewed changes

eskimor approved these changes Jan 8, 2021

View reviewed changes

drahnr approved these changes Jan 8, 2021

View reviewed changes

Update node/overseer/src/lib.rs

9e94b09

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

rphmeier closed this Jan 8, 2021

rphmeier added 12 commits March 27, 2021 17:33

Merge branch 'master' into rh-messaging-refactor

efd00d2

give every subsystem an unbounded sender too

23115b5

remove metered_channel::name

c4a7a8e

1. we don't provide good names 2. these names are never used anywhere

unused mut

0d4115a

remove unnecessary &mut

fe43ae5

subsystem unbounded_send

329dddc

remove unused MaybeTimer

42174c0

We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.

remove comment

6ba8f33

split up senders and receivers

8fd7ce6

update metrics

ccbd091

fix tests

ef5dca8

fix test subsystem context

8b50bc2

rphmeier reopened this Mar 27, 2021

Merge branch 'master' into rh-messaging-refactor

984fb9f

ordian reviewed Mar 27, 2021

View reviewed changes

node/overseer/src/lib.rs Outdated Show resolved Hide resolved

ordian reviewed Mar 27, 2021

View reviewed changes

node/subsystem/src/lib.rs Show resolved Hide resolved

rphmeier mentioned this pull request Mar 28, 2021

Split NetworkBridge and break cycles with Unbounded #2736

Merged

rphmeier added 2 commits March 28, 2021 03:45

fix flaky test

8bfe645

fix docs

832686f

ordian reviewed Mar 28, 2021

View reviewed changes

node/overseer/src/lib.rs Outdated Show resolved Hide resolved

rphmeier added 2 commits March 28, 2021 14:52

doc

efb3633

use select_biased to favor signals

7baeca0

ordian reviewed Mar 28, 2021

View reviewed changes

node/subsystem/src/lib.rs Outdated Show resolved Hide resolved

rphmeier and others added 3 commits March 28, 2021 16:35

Update node/subsystem/src/lib.rs

0ae3449

Co-authored-by: Andronik Ordian <[email protected]>

Merge branch 'master' into rh-messaging-refactor

7171e05

Merge branch 'master' into rh-messaging-refactor

dd117ce

eskimor approved these changes Mar 28, 2021

View reviewed changes

ghost merged commit 322ccd0 into master Mar 28, 2021

ghost deleted the rh-messaging-refactor branch March 28, 2021 15:55

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overseer: subsystems communicate directly #2227

Overseer: subsystems communicate directly #2227

rphmeier commented Jan 8, 2021 •

edited

Loading

eskimor commented Jan 8, 2021

coriolinus left a comment

coriolinus Jan 8, 2021

eskimor Jan 8, 2021

rphmeier Jan 8, 2021

coriolinus Jan 8, 2021

eskimor left a comment

drahnr left a comment

ordian Mar 28, 2021

rphmeier Mar 28, 2021

rphmeier Mar 28, 2021

rphmeier commented Mar 28, 2021

ghost commented Mar 28, 2021

Overseer: subsystems communicate directly #2227

Overseer: subsystems communicate directly #2227

Conversation

rphmeier commented Jan 8, 2021 • edited Loading

eskimor commented Jan 8, 2021

coriolinus left a comment

Choose a reason for hiding this comment

coriolinus Jan 8, 2021

Choose a reason for hiding this comment

eskimor Jan 8, 2021

Choose a reason for hiding this comment

rphmeier Jan 8, 2021

Choose a reason for hiding this comment

coriolinus Jan 8, 2021

Choose a reason for hiding this comment

eskimor left a comment

Choose a reason for hiding this comment

drahnr left a comment

Choose a reason for hiding this comment

ordian Mar 28, 2021

Choose a reason for hiding this comment

rphmeier Mar 28, 2021

Choose a reason for hiding this comment

rphmeier Mar 28, 2021

Choose a reason for hiding this comment

rphmeier commented Mar 28, 2021

ghost commented Mar 28, 2021

rphmeier commented Jan 8, 2021 •

edited

Loading