RFC: Extend STORE protocol to give meta information about capabilities in-protocol

[oskarth]

Problem

STORE protocol is symmetric, and just knowing a peer is using STORE doesn't tell you much about what they are storing, if anything. The node can primarily be a "server" node or a "client" node.

Proposed solution

There are a few different dimensions of information that are desirable:

• Are they persisting messages at all? For which topics? (e.g. default pubsub topic to start with)
• For how long are they store? (7d retention policy e.g.)
• For how long have they been online (e.g. online window in epochs, assuming honest => extend with checks)

This informs a peer with enough data to know if:

• they should ask some other peer
• they should ask more than one peer

Rought sketch for protobufs:

RPC {
  HistoryQuery
  HistoryResponse
  MetaRequest
  MetaResponse

Acceptance criteria

• Rough consensus
• Raw spec

Notes

Adaptive node discussion context: #87

Additionally do similar for FILTER.

cc @staheri14 @jm-clius

also fyi @cammellos re core pov

[D4nte]

Alice needs to do one history request to Bob to know whether Bob stores the pubsubTopic she is interested in. How would this initial request look like?

As /vac/waku/store/2.0.0-beta2 is meant to retrieve historical data, it may make more sense to have a separate protocol to exchange information about storage capabilities such as /vac/waku/store/2.0.0-beta2/info or /vac/waku/store/2.0.0-beta2/identify.

Also, you make a good point in the problem section:

STORE protocol is symmetric, and just knowing a peer is using STORE doesn't tell you much about what they are storing, if anything. The node can primarily be a "server" node or a "client" node.

peer is using STORE

What does this mean in practical term? I assume that if Bob is using STORE it means that he includes /vac/waku/store/2.0.0-beta2 as part of his identify response to Alice.

IMHO, if Alice does not store any data but only supports the STORE protocol as a querying node then I don't believe she should advertise that she supports /vac/waku/store/2.0.0-beta2.

When Alice connects to Bob:

• Alice does not sends /vac/waku/store/2.0.0-beta2 as part of identify
• Bob sends /vac/waku/store/2.0.0-beta2 as part of identify.

From this point, it is clear to Alice and Bob who can query the other one for historical data.
I do not believe (but I'd need to dig deeper in the identify protocol to confirm) that Bob needs to know whether Alice supports the store protocol at any point in time.

Finally, if down the line we use pubsubTopics as a way to shard the network, then it may be critical for Alice to know upfront if Bob is part of her shard.
In this case, then the pubsubTopic could be appended to the protocol string advertised by Bob, e.g. /vac/waku/store/2.0.0-beta2/topic/waku/default so that from the moment the identify protocol is done, Alice know whether she should use Bob as a store server or drop the connection.

[staheri14]

The solution looks good to me @oskarth! such info re nodes capabilities is also necessary for the FT store protocol where nodes want to retrieve message history from a peer who has been online for a certain time window.

I have some suggestions:
The current solution assumes that two nodes are randomly connected and have no prior information about each others' resources and capabilities. This somewhat implies that a querying node has do a linear search (linear in the total number of nodes) and run the meta request/response protocol to find out a suitable candidate to query historical messages from.
Aa an alternative and perhaps a more efficient solution we may want to consider an approach where nodes periodically advertise their capabilities over a designated pubsub topic . This way, a requestor node connects to another node with the knowledge that the queried node is capable of responding to that request. Moreover, the advertised capability messages can also be persisted in the store db.
The capability messages can contain the same fields as in the meta request i.e.,

Are they persisting messages at all? For which topics? (e.g. default pubsub topic to start with)
For how long are they store? (7d retention policy e.g.)
For how long have they been online (e.g. online window in epochs, assuming honest => extend with checks)

[oskarth]

Great comments! Thanks both. Sounds like we have to do a bit more thinking and investigation on this one to come up with a good synthesis.

[jm-clius]

I do not believe (but I'd need to dig deeper in the identify protocol to confirm) that Bob needs to know whether Alice supports the store protocol at any point in time.

This may fall outside the domain that Waku should have influence over - from libp2p's perspective, if a peer has a certain protocol mounted that protocol should be advertised in the identify message. I think, however, we can use a similar pattern to what you suggested for topics, by appending the server/client/both information to the mounted protocol string.

This is not yet clear in my mind, but I'd propose something that will keep "protocols" and "capabilities" semantically separated. A protocol is a type of capability, but "capabilities" is a broader concept. We could even distinguish between "configured" capabilities, e.g. if a node is configured as a store client or server or both, and more "transient" capabilities, e.g. how much history the node is actually keeping, history topics, etc. "Configured" capabilities could be seen as protocol extensions and appended to the protocol string, while "transient" requires a capability negotiation or further discovery process as suggested.

[D4nte]

from libp2p's perspective, if a peer has a certain protocol mounted that protocol should be advertised in the identify message

The specs says This is a list of protocols supported by the peer. Please note I got some clarification from libp2p folks: libp2p/specs#324 (comment)

[oskarth]

Turning this into a discussion instead.

an alternative and perhaps a more efficient solution we may want to consider an approach where nodes periodically advertise their capabilities over a designated pubsub topic . This way, a requestor node connects to another node with the knowledge that the queried node is capable of responding to that request. Moreover, the advertised capability messages can also be persisted in the store db.
The capability messages can contain the same fields as in the meta request i.e.,

Discussed briefly with Hanno and think this seems reasonable. Basically this acts as an event log, and then you can ask a store node for this information.

We can also complement this with an aggregator/view/log compact function, so you can ask for latest state of capabilities for each node. Basically for node A B C announcing capabilities Acap0 Bcap0 Acap1 Cap0 Bcap1 Acap2 on pubsub topic "capabilities", then a new node can send protobuf message getNodesWithCapabilties(<optional capabilities filter>) => {A: Acap2, B: Bcap1, C: Cap0} or similar.

What do people think of this?

[D4nte]

With this proposal, I believe you are moving toward a discovery domain issue which seems to be the next step of the original problem:

This informs a peer with enough data to know if:

• they should ask some other peer
• they should ask more than one peer

Meaning, the proposal helps answer the question "How do I find some other peer?".

At this stage I am not really sure how the sequence would work, from bootstrapping to querying the right store node. Most likely because we currently bootstrap by directly connecting to fleet nodes that have store support which does not fit really well with the broadcasting capability solution.

Which makes me wonder, is this too early? If we think the best way to manage store server metadata capability is by broadcasting said capability on the network to facilitate discovery of the right store node, then it means we should resolve the node discovery problem first, design it and test it on the field before trying to resolve the store metadata problem.

[oskarth]

You are right it touches on discovery domain. The way I think of it, it essentially works at multiple layers.

1. First, you connect to some bootstrap nodes that lets you connect to some nodes in the network
2. Then, you use something like DNS based discovery or Discovery v5 to find more nodes in the general network

• @jm-clius is working on a proposal for this, starting with DNS based discovery. You can do some namespacing of domains etc, and theoretically with Discv5 topics you could do more capabilities here (in practice topic based discovery seems to not work well, or it has issues with needle-in-haystack capabilities, which is what we'd fall under, especially wrt Disc v5 due to Eth network size). But basically you'd mostly get "more nodes" in general here, not more fine grained info.

3. Then you would need to find nodes by specific capabilities, and this can be dynamic (e.g. store this topic, or online for this window etc) <-- that's where proposal fits in
4. Some capabilities might only be discovered in-protocol upon handshake, say. This is where OP was going, initially

This roughly reflects previous discussions in this area re capabilities and discovery

[D4nte]

Some capabilities might only be discovered in-protocol upon handshake, say. This is where OP was going, initially

I understood the "Broadcast" proposal as the only way to communicate capabilities. It seems you are hinting we could have metadata capabilities communicated when connecting to a store node AND capabilities broadcasted on the network under specific pubsub topic by store node. Is my understanding correct?

[oskarth]

I'm saying it a spectrum and different things can make sense here depending on the specific use case.

In #374 (comment) I'm only talking about using a pubsub topic to communicate capabilities (with the aggregator idea as a potential addon).

[oskarth]

Generic message follows

I'm going to go ahead and close this discussion as:

1. It is quite old and we are moving away from GH discussions (in favor of either forum.vac.dev or normal issues)

2. Existing discussion has evolved and is captured in more targeted issues already

If you feel like this is in error, please create a thread on Vac forum or a new issue.