refactor: ⎆ Migration to pull-streams (from Node.js Streams)

[dignifiedquire]

As noted in #362 I did the due diligence of confirming that using pull-streams all the way through solves our issues with secio. The result of this can be seen in libp2p/js-libp2p-switch#67. (Warning lots of module linking required to get this running)

The next step is to move from the prototype into a fully tested version for all modules.
Below is a list of the modules that will need work.

ipfs

• js-ipfs - PR [WIP] Move to pull-streams #440
• interface-ipfs-core - PR fix(files): don't expect a specific sorting of files ipfs-inactive/interface-js-ipfs-core#70
• js-ipfs-api - delayed - Issue Migrate to pull-streams ipfs-inactive/js-ipfs-http-client#367
• js-ipfs-unixfs-engine - PR [WIP] Pull streams ipfs-inactive/js-ipfs-unixfs-engine#64
• js-libp2p-ipfs (Only testing) - PR pull ipfs-inactive/js-libp2p-ipfs-nodejs#17
• js-libp2p-ipfs-browser (Only testing) - PR refactor: use pull-streams ipfs-inactive/js-libp2p-ipfs-browser#84
• js-ipfs-merkle-dag - PR [WIP] refactor: move to pull-streams js-ipfs-merkle-dag#44
• js-ipfs-block-service - PR refactor: Migrate to pull-streams js-ipfs-block-service#23
• js-ipfs-ipld - PR refactor: migrate to pullstreams ipld/js-ipld#54
• js-ipfs-repo - PR [WIP] Move to pull-streams js-ipfs-repo#76
• New interface-pull-blob-store (replaces abstract-blob-store) https:/ipfs/interface-pull-blob-store
• New idb-pull-blob-store https:/ipfs/js-idb-pull-blob-store
• New fs-pull-blob-store https:/ipfs/js-fs-pull-blob-store
• js-ipfs-bitswap - PR [WIP] Pull streams js-ipfs-bitswap#25
• js-multistream - PR [WIP] pull-streams multiformats/js-multistream-select#19

libp2p

• interface-connection - PR Migrate to pull-streams libp2p/interface-connection#10
• js-libp2p-identify - PR [WIP] move to pull-streams libp2p/js-libp2p-identify#2
• js-libp2p-secio - PR [WIP] Refactor to use pull streams libp2p/js-libp2p-secio#8
• js-libp2p-spdy - PR [WIP] Move to pull-streams libp2p/js-libp2p-spdy#16
• js-libp2p-websockets - PR [WIP] Move to pull-streams libp2p/js-libp2p-websockets#18
• js-libp2p-tcp - PR Migrate to pull-streams libp2p/js-libp2p-tcp#20
• js-libp2p-webrtc-star PR Migrate to pull-streams libp2p/js-libp2p-webrtc-star#12
• js-libp2p-ping - PR migrate to pull-streams libp2p/js-libp2p-ping#11
• js-libp2p-swarm - PR [WIP] Add secio & move to pull-streams libp2p/js-libp2p-switch#67
• interface-stream-muxer - PR [WIP] Pull-streams libp2p/interface-stream-muxer#7
• interface-transport - PR https:/libp2p/interface-transport#8

Documentation

• Add a note about pull-streams to every readme
• Create a FAQ document collecting all important information on how to use pull-streams, how to convert them, why we use them etc.

[daviddias]

Thank you for captaining this @dignifiedquire :) :) Let's make sure not to forget documentation :)

[dignifiedquire]

Draft for the readme note.

This module uses pull-streams for all stream based interfaces. You can find details of why we use them in this document. If you need to convert them to node-streams, you can do so easily with pull-stream-to-stream like this:

const toStream = require('pull-stream-to-stream')
const pullStream = getPullStream()
const myNodeStream = toStream(pullStream)

[jbenet]

before doing the work of chaning every module, i highly recommend you pick a small set and work with them-- see how they go. You wont understand how an abstraction fits a project until you play with it for a while. and def better to find that out with a small set than after changing everything.

[dignifiedquire]

@jbenet we already did that as outlined at the beginning of this issue. I ported the minimal needed set to get secio in swarm working and see how things are working out, and they worked out great. Afterwards I talked with @diasdavid and we decided to move forward. The result of this is this endeveaour of moving all things into high quality ports.

[RichardLitt]

Sub. Let me know if I can help with READMEs.

[dignifiedquire]

Thanks @RichardLitt ❤️ will ping you as soon as I have a consistent message ready.

[daviddias]

Missing from the list:

• js-multistream

[dignifiedquire]

Missing from the list:

js-multistream

added

[daviddias]

One more: libp2p/interface-transport#8

[daviddias]

Status Update - The enchanted adventure of the missing backpressure mechanism.

The migration to pull-streams is almost complete, we currently have PR's across every single module that required to be updated in order for the migration to be complete. These PR include the code, documentation is still lacking.

There is one major blocker though, although we've been sucessful porting almost every single module to pull-streams, our stream-muxing stress tests are failing (fortunately we had them before this endeavour). We are still investigating the issue, having ruled out a bunch of the possible causes, we now have a strong conviction that the backpressure mechanism is failing when we convert back and forth pull and Node.js streams.

Me and @dignifiedquire have been spending a considerable amount of time debugging, if you would like to help, I've created 3 PR with different experiments and have added a way to test it, so you can observe the failures by yourself. All of these are documented:

• Migration to Pull Streams - Experiment A libp2p/js-libp2p-spdy#19
• Migration to Pull Streams - Experiment B libp2p/js-libp2p-spdy#20
• Migration to Pull Streams - Experiment C libp2p/js-libp2p-spdy#21

	tests	stress tests	mega stress tests
Mode 0 (Current implementation, just Node.js Streams	✔️	✔️	✔️
Mode A (Wrap pull streams into Node.js Streams)	✔️	𝙓 (passes if throttled)	𝙓 (and OOM)
Mode B (Using Node.js Streams for transport)	✔️	✔️	OOM
Mode C (Mimics real use)	✔️	𝙓 (passes if throttled)	𝙓 (and OOM)

In addition to the fact that (apparently), backpressure stops working properly when too much wrapping happens, we also see the process running Out Of Memory when tested exhaustively. This can be because the backpressure mechanism doesn't work as it should (keeping a lot of memory floating around) or simply because pull-streams have a high memory fingerprint, making the memory run out just because of having too many instances (less probable).

[dignifiedquire]

I finally had a breakthrough just now! I figured out why the pausing happens. The reason for it is that at some point the pull-stream side actually has too much data, and says "hey slow down" to the spdy-transport stream (the framer to be exact). This is happens via the .write call returning false (https:/pull-stream/pull-stream-to-stream/blob/master/index.js#L36).

After this everything comes to a halt quickly due to the fact that the framer is never resumed.

The only somewhat viable solution I found so far was to add add

if (this.isPaused()) {
  this.resume()
}

to the read call in in the (Scheduler.prototype._read](https:/indutny/spdy-transport/blob/master/lib/spdy-transport/protocol/base/scheduler.js#L91) which is the base class of the framer. The explanation as to why this makes sense, is that the framer is piped to the underlying transport. In the case the underlying transport exerts backpressure onto the framer it calls .pause. When it is ready to continue writing it will call .read, triggering the ._read method on the source, i.e. this is the sign that the socket is ready to receive more data and as such the data flow can be resumed.

I am not certain in any way that this is the correct fix, so I'm very happy to receive more input here. One important step I have not done yet is to create a test case using only node streams to recreate this problem with backpressure on the underlying socket with spdy-transport.

[dignifiedquire]

I've created the above mentioned test and fix here: dignifiedquire/spdy-transport@0e9c847

[daviddias]

@dignifiedquire where are the PR's for:

- [ ] js-libp2p-ipfs (Only testing)
- [ ]  js-libp2p-ipfs-browser (Only testing)

[daviddias]

@dignifiedquire are there PR for the ipld-service?

[dignifiedquire]

@diasdavid no there is not, I would suggest changing that when we actually integrate it.

[daviddias]

@dignifiedquire we've been directing some groups to check it out, we have to make sure it continues to work

[dignifiedquire]

we've been directing some groups to check it out

We did? Sure if people are actually using it we can update it now. I did not expect that

[daviddias]

@dignifiedquire yep :)

I've made my CR to all the remaining repos, once all of those points are taken care (very little stuff), I'll do the merge and release \o/ :D

[daviddias]

Woooo! All the Pull Streams are in :D