New RPC system

[keks]

There's a few issues with the http API and it would be cool to support different kinds of transports like Unix domain sockets or websockets. So having a protocol that runs atop any reliable transport would be desirable.

This issue is about exploring the problem space and having a place to put ideas that need to be considered when we finally start working on this.

Relevant issues:

• Wrap results in something indicating the type of the result #31 Make sure we can distinguish error from value at all times
• Support Unix Sockets kubo#4218 Add support for Unix domain sockets

[magik6k]

Sending cbor rpcs both ways should be good enough.

CBOR seems to be perfect for RPC over reliable transports as it supports streaming, has defined boundaries and is very compact when data structure is designed well.

I'm not sure about the exact requirements, but what would probably work well would be client sending requests with some sort of unique identifier in cbor frames, and then the server would respond to those requests, not necessarily in the order they came, possibly with multiple responses over time (commands like ipfs ping). Only problem I see here is figuring out backpressure.

[keks]

While I agree that CBOR seems like a good fit, I'd probably advocate using multistream for this.
But yeah, CBOR would probably be the default format.

re back pressure: don't most reliable protocols handle that? At some boint the receive buffer is full and the client does acknowledge further packets. We can call TCPConn.SetReadBuffer(somethingSmall) to make sure the sender doesn't send too much, but I'm not sure how to find a good value here. Unix domain sockets have hard backpressure, the sender won't be able to write when the receiver doesn't read.

[Stebalien]

While I agree that CBOR seems like a good fit, I'd probably advocate using multistream for this.
But yeah, CBOR would probably be the default format.

YES! Multi all the things.

Only problem I see here is figuring out backpressure.

I'm not sure this is necessary. This is a privileged, local interface.

---

We may also be able to get away with using an existing RPC system.

[magik6k]

I was thinking about throwing multistream at it, one problem is that adds 1rtt (even if local this has some effect on short lived sessions, though it can be probably turned into 0rtt with some clever hacks), another is the added complexity - API wrappers will have to implement multistream handshake. Advantage is that we could probably reuse API port and try to detect if the incoming request is HTTP or multistream.

We may also be able to get away with using an existing RPC system.

That would be optimal

[dignifiedquire]

Some thoughts, as I have been thinking about this as well.

• Adding multistream is quite a large requirement if you want to make it easy for other clients to connect to.
• List of transports I would like to see are
  • unix sockets
  • windows pipes
  • websockets
  • in process
• CBOR is great but does have overhead, so I think there should be ways to avoid this for certain operations, i.e. streaming a file shouldn't go through cbor, but rather should be piped directly through with the most minimal overhead possible.

[dignifiedquire]

As reference, the RPC api from ethereum has quite a nice implementation of the various transport options here: https:/ethereum/go-ethereum/tree/master/rpc. (this does not mean I am suggesting to use a json rpc api).

I think it is important to keep in mind that this would be an end user interface ideally, an the CLI just one of many clients, so the requirements should be kept as simple as we can on the client side. Which makes me think using any sort of multistream would be too much overhead there.

[Kubuxu]

Other option could be capnproto https://capnproto.org/.

I would love to see the API in RPC form with pressure on performance. As an example, possible use case could be FUSE.

IMO it could mimic core-api defined in go-ipfs. As far as I know JS's interface-core-api closely mimics current HTTP api and so the current CLI.
We want to rewrite current commands to use go's core-api so it would be great if later we could just plug'n'play it over the RPC.

---

streaming a file shouldn't go through cbor, but rather should be piped directly through with the most minimal overhead possible.

This would require muxer if there are to be multiple concurrent requests from one client (something we most certainly want). GRPC provides it but mostly because it is over QUIC so they get it for free.

[Stebalien]

Really, it would be kind of nice to be able to use libp2p. That would also make remote administration easier. @vyzo actually just wrote a dead-simple unix socket based transport for libp2p but we could use any transport, really. The client doesn't have to implement the entire libp2p stack, just the individual transport (and probably multistream-select but that's quite simple).

[keks]

Let me collect design constraints we've identified so far:

• Errors should always be distinguishable from values
  • Corollary: values should always be identified as such
  • we have a situation where we request a raw object and that may look like an error and we don't can't know whether we correctly received the stored object that looks like an error or whether we actually received an error...
• Web friendly
• Add support for UNIX sockets
• Keep an eye on performance
• make it easy for clients to use (i.e. to implement the API transport protocol)
• Allow multiple long-running commands over a single connection
  • this requires muxing (with backpressure on individual streams)
• don't reinvent the wheel

---

My main concern is that requiring the client to mux multiple command call streams over a single connection is decidedly not easy to implement, unless you take something off-the-shelf (QUIC, HTTP/2, yamux(?)). However, if we decide to go for one of these, I guess we already are at 60% of basic libp2p, so maybe it does make sense to just use that.

[Kubuxu]

Allow multiple long-running commands over a single connection
this requires muxing (with backpressure on individual streams)

This isn't true if you go with RPC and handles. Then you can have multiple long running commands. Example imagine a Read on file.

The way you are thinking about is just streaming the file. In this case, muxing is required.

Another way is to give the user a handle for the open file, then user requests a 2KiB chunk of the file. You prepare a buffer on the server side and after it is filled to 2KiB you send it over with minimal framing referencing the handle you've given.

This way there is no blocking but implementation itself is more complex. We have to decide if that complexity is worst than using a muxer.

[Stebalien]

Yeah, muxing complexity could be a concern but I think we can just make it a performance tradeoff. We can expose the API over as many protocols as we want so users can use connection pools with no muxers.

Requirement: web friendly.

[dignifiedquire]

as much as I like libp2p, adding the requirement of loading all of js-libp2p into the browser sounds like a very high overhead for an important use case. And makes it much less attractive at least for me

…

On 3. Oct 2018, 02:17 +0200, Steven Allen ***@***.***>, wrote: Yeah, muxing complexity could be a concern but I think we can just make it a performance tradeoff. We can expose the API over as many protocols as we want so users can use connection pools with no muxers. Requirement: web friendly. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[keks]

@Kubuxu I'm not convinced. Imagine the daemon does not yet have the file that the user requested and needs to get it from the ipfs network. That will take a while. Should the RPC server just not respond until it has at least the beginning of the file and block all pending requests? Should it return "I requested it but don't have it yet, ask me later"? At the end of the day, that approach just let's us end up polling the server for new data, or the connection blocks until the server has the response. Both options lead to a terrible experience for the API implementor.

@dignifiedquire I'm also not convinced using libp2p is too much of a dependency, but @Stebalien claimed that

The client doesn't have to implement the entire libp2p stack, just the individual transport (and probably multistream-select but that's quite simple).

Do you agree with that sentence? If so, we might be able to use a subset of libp2p. For example, my understanding is that libp2p allows the responder to do RPC calls on the initiator. We wouldn't need that kind of functionality, so we can just ignore that part while staying in the libp2p protocol.

---

One more observation: If we want to be web-friendly, we can choose between using HTTP directly, or speaking any protocol on top of a websocket. Going through the list of constraints, I realize HTTP/2 satisfies any constraint on the transport protocol we've set so far. It's a widespread standard, has implementations in just about every language out there (haven't checked APL ;), is web-friendly (I think all browsers support it nowadays?) and it handles stream muxing for us.
Also, it is pretty fast and allows for optimizations using push promises, and important to me personally, we don't reinvent the wheel.

Thus, I suggest using HTTP/2 as the foundation of the new API protocol and we start talking about the request and response message formats. These have been causing issues in the current API (i.e. it's difficult to tell a value from an error, or generally which type it is - compare #73). Basically we need to design how we map the RPC interface on HTTP/2, because it does give us some more possibilities than HTTP/1.1 and our current mapping is terrible.

[dignifiedquire]

he is right we don’t have to put in everything, but I am not convinced it isn’t still too much

…

On 3. Oct 2018, 12:18 +0200, keks ***@***.***>, wrote: @Kubuxu I'm not convinced. Imagine the daemon does not yet have the file that the user requested and needs to get it from the ipfs network. That will take a while. Should the RPC server just not respond until it has at least the beginning of the file and block all pending requests? Should it return "I requested it but don't have it yet, ask me later"? At the end of the day, that approach just let's us end up polling the server for new data, or the connection blocks until the server has the response. Both options lead to a terrible experience for the API implementor. @dignifiedquire I'm also not convinced using libp2p is too much of a dependency, but @Stebalien claimed that > The client doesn't have to implement the entire libp2p stack, just the individual transport (and probably multistream-select but that's quite simple). Do you agree with that sentence? If so, we might be able to use a subset of libp2p. For example, my understanding is that libp2p allows the responder to do RPC calls on the initiator. We wouldn't need that kind of functionality, so we can just ignore that part while staying in the libp2p protocol. One more observation: If we want to be web-friendly, we can choose between using HTTP directly, or speaking any protocol on top of a websocket. Going through the list of constraints, I realize HTTP/2 satisfies any constraint on the transport protocol we've set so far. It's a widespread standard, has implementations in just about every language out there (haven't checked APL ;), is web-friendly (I think all browsers support it nowadays?) and it handles stream muxing for us. Also, it is pretty fast and allows for optimizations using push promises, and important to me personally, we don't reinvent the wheel. Thus, I suggest using HTTP/2 as the foundation of the new API protocol and we start talking about the request and response message formats. These have been causing issues in the current API (i.e. it's difficult to tell a value from an error, or generally which type it is - compare #73). Basically we need to design how we map the RPC interface on HTTP/2, because it does give us some more possibilities than HTTP/1.1 and our current mapping is terrible. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[keks]

@Stebalien @Kubuxu @diasdavid @dignifiedquire

What is your opinion on using HTTP/2 directly, without a generic layer of abstraction such as libp2p? It seems to satisfy all of our requirements, and if we go for something else we'll end up using a muxing protocol over websockets over http/2 (because soon everything will be http/2), which seems ridiculous to me.

Note that go's HTTP/2 library is not affected by ipfs/kubo#5168 (server has to drain request body before writing), because golang/go#15527 only seems to affects HTTP/1.x.

This does not have to be a fixed decision, but it would set the direction for research and experimentation in the coming weeks.

[Kubuxu]

@Kubuxu I'm not convinced. Imagine the daemon does not yet have the file that the user requested and needs to get it from the ipfs network. That will take a while. Should the RPC server just not respond until it has at least the beginning of the file and block all pending requests? Should it return "I requested it but don't have it yet, ask me later"?

Think about it as a message protocol. You ask me for something, I don't have it yet, I just won't respond to you yet. One I do have it, I will send you response referencing request handle. Until then you can make other requests.

No blocking, no polling. Unless you mean waiting on a socket as polling but that is always a case.

[keks]

@Kubuxu you're right, that would work. However, that sounds like a simple muxing protocol to me. And yes, it has backpressure, but at a huge price: After requesting a file, the server will at some point send a "data ready" message, and the client can request it, block by block. However, this introduces a round-trip time between every block. Even though we will probably use this protocol mostly on localhost, this still is unnecessary overhead.
Why should we build the protocol ourselves? We would have even more stuff to maintain, and others would also need to implement it to speak with the server, making it a bigger job.

IMHO a DIY muxing protocol is ruled out by the "don't reinvent the wheel" requirement - unless we find an important aspect where a DIY muxing protocol is superior.

[Stebalien]

@dignifiedquire, @keks you're probably right. I'd like to additionally support libp2p as a transport but we'll need to make this work over other transports as well. However, I'd like to make sure it's efficient over some libp2p transport.

[Kubuxu]

@keks this is how you would handle this problem in case of most RPCs.

buf := make([]byte, 32<<10)
hd := rpc.Open("/ipfs/Qm...AAA")
_, err := hd.ReadCtx(ctx, buf)

This is how I would expect it to look. In the case of capnproto (https://capnproto.org/rpc.html) , there would be no additional roundtrip delay but in case of other protocols, there could be one.

[Kubuxu]

It isn't DIY muxing protocol it is just handling multiple requests over RPC. gRPC with its pipes is one of the few (if not the only one), that uses muxing internally.

---

There might be also miss-understanding here (probably on my part). I am thinking about a protocol for transporting coreapi out of the process (I think I was linked here from OKR about that). I think you are working out new http-api transport.
Sorry if that is the case.

[keks]

@Kubuxu right, you can reduce roundtrips significantly with pipelining. However in that case, we need to schedule the requests in some way, such that none of our requests starve or grow a long queue because they process slower than the data comes in. This in itself probably is a pain to build - and already is part of HTTP/2 (the credit-based flow control part).

And again, you failed to make a point why we shouldn't use HTTP/2. In the web context we would run capnproto over websocket over HTTP/2, just to let capnproto do something (but not as good) that HTTP/2 already does. I assume there is a reason you don't want to use HTTP/2, otherwise you wouldn't suggest using something different. I would be happy to head about that reason.

---

We are looking for a protocol to replace the current HTTP API. We did not limit ourselves to HTTP, but given the web-friendly requirement it's an obvious candidate.

[Kubuxu]

In terms of HTTP API I don't have anything against it. If it is supposed to be accessible to web technologies HTTP/2 is probably the best for it (there isn't much that can be used for it).

In terms of universal coreapi:

1. it is a huge requirement (HTTP/2 is huge in terms of code and complexity and there are languages without implementation)
2. it has high overhead for short-lived connections (relatively)
3. crypto overhead (HTTP/2 forces SSL) in case of just ChaCha20 (which is the best case scenario). I am getting ~400MB/s per core. This is alright for web but not for generic high-performance api.

In general:

1. self-signed certificate troubles

[Stebalien]

We shouldn't rely on any specific transport. Instead, we can just rely on some specific features of the transport.

[keks]

In ipfs/kubo#5576 a user noted that ipfs add responds with objects that contain the file size as string and suggested using ints instead. I agree this is better, but noted that backward compatibility prevents us from just changing it.

I was thinking that we might want to change this with the new API protocol (because cleaning up the API seems like a good idea when you are not limited by backwards compatibility requirements, but realized that if we want to use exactly the same map[string]Command, we would also reuse the type information. So either we instantiate two map[string]Command, or we clutter the Command struct with instructions for both APIs. I like neither option, but maybe we'll find another way.

But first: Do you think we should tackle that class of issues in this effort at all?

[Kubuxu]

ipfs add responds with objects that contain the file size as string.

This was probably done due to limited number precision in JS when decoding numbers from JSON.

[keks]

We had a session on this today.

We realized that we have too many targets we want to optimize for, so we won't have a one-size-fits-all solution. For example, it's really hard to get both great performance and web-friendliness.
Instead, we want to use the current CoreAPI interface definition and use that to generate a machine-readable interface definition language (IDL), that can then be used to compile to most of the boilerplate code for different RPC mechanisms. Possible mechanisms we have in mind (not a definitive list):

• HTTP (for browsers)
• libp2p
• something on top of UNIX sockets, so we can send file descriptors (maybe even libp2p)

Initially I envisioned that we could have a general CoreRPC interface that we could implement for each RPC protocol, and then build upon that once to implement the CoreAPI interface. During the discussion we realized that probably RPC mechanisms are too different to actually have one such interface, so we instead want to use the IDLs to reduce the burden of implementing CoreAPI from scratch for each RPC system.

Regarding the IDL: I'll do some research about what is already out there and see if any of these has what we need. The requirements on the IDL that we identified are a proper type system that allows us to specify types like CIDs, files and streams.

We also spent some time discussing the negotiation of API versions on the RPC. We decided that we again only need to have major versions as numbers, and we'll be backwards-compatible with regard to these, but we will (a) provide a command that allows the client to fetch the IDL the server is targetting and (b) when the client supplies flags the server does not understand, it will send an error that contains both an error message and the cid of the server's IDL.

The raw notes of this discussion (and others) are here: https://cryptpad.fr/code/#/2/code/edit/vFqT8IRFXfqHkCWvY9CKsbEA/

ping @Kubuxu @Stebalien @michaelavila (a sample of the attendees) is that what you heard in the disussion?

[keks]

@dignifiedquire I think you may have objections wrt. client implementation complexity, but we'll try to keep this down for the HTTP version. Does that sound good to you?

[lanzafame]

@keks

Instead, we want to use the current CoreAPI interface definition and use that to generate a machine-readable interface definition language (IDL), that can then be used to compile to most of the boilerplate code for different RPC mechanisms.

Would I be correct in thinking that what we are after is gRPC (language-agnostic RPC with IDL for code generation) plus added support for the RPC to be transport-agnostic and the IDL to support code generation for that as well?