libp2p-webrtc-star + bitswap throws TypeError: Message too large (can send a maximum of 65536 bytes)

[wehriam]

• Version:

  • js-ipfs version: 0.36.3-
  • Repo version: 6
  • System version: x64/darwin
  • Node.js version: v10.15.3

• Platform:

  • Darwin jordan-2.local 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
  • Google Chrome, Version 74.0.3729.169 (Official Build) (64-bit)

• Subsystem:

  • bitswap
  • libp2p-webrtc-star
  • simple-peer

Type:

• Bug

Severity:

• Medium

Description:

When sharing a series of files (an HLS video stream) among a single go-ipfs host and 2+ js-ipfs clients, simple-peer reports that messages are too large.

bitswap:network:QmSehzzL:error Error: TypeError: Message too large (can send a maximum of 65536 bytes)
at makeError (http://localhost:3000/cluster/main.js:231317:13)
at Peer../node_modules/@bunchtogether/bolt-client/node_modules/simple-peer/index.js.Peer._write (http://localhost:3000/cluster/main.js:230801:27)
at doWrite (http://localhost:3000/cluster/main.js:233268:64)
at writeOrBuffer (http://localhost:3000/cluster/main.js:233257:5)
at Peer../node_modules/@bunchtogether/bolt-client/node_modules/simple-peer/node_modules/readable-stream/lib/_stream_writable.js.Writable.write (http://localhost:3000/cluster/main.js:233175:11)
at http://localhost:3000/cluster/main.js:244915:30
at http://localhost:3000/cluster/main.js:214036:9
at http://localhost:3000/cluster/main.js:210771:16
at http://localhost:3000/cluster/main.js:214036:9
at http://localhost:3000/cluster/main.js:214036:9 +2ms

Steps to reproduce the error:

1. Connect a go-ipfs node to two js-ipfs nodes via Websocket
2. Connect the js-ipfs nodes to each other via libp2p-webrtc-star
3. Request a sequence of unique 2 MB files from the go-ipfs server

I have been unable to reproduce this outside of my test environment. I have attempted to reproduce at https://jsfiddle.net/dqzouyj1/2/

[wehriam]

@dirkmc - As mentioned earlier in my comments on your garbage collection PR, I'm implementing a video streaming application. Unfortunately the transfer rate between WebRTC peers - even on a LAN - ends up being really slow, often dropping to double digit Kbps despite direct connections between peers with all of the blocks.

I'm wondering if this bitswap error is to blame? I'm still learning pull-streams so I can't identify where the relevant chunking occurs.

As always, thanks for your awesome work and let me know if there's anything I can do to assist.

[dirkmc]

@wehriam my guess is that bitswap is attempting to send a message that is larger than the maximum size allowed by the underlying transport. That error is probably being logged out here.

@vmx is it possible bitswap is attempting to send blocks that are bigger than 65536 bytes?

[dirkmc]

I wonder if it's coming from this? muaz-khan/RTCMultiConnection#606

[dirkmc]

@vasco-santos do you think the above error may be coming from js-libp2p-webrtc-star?

[vasco-santos]

I haven't seen a similar error before, but taking into account the setup description I would say that it may come from the webrtc-star module.

However, according to the fiddle that @wehriam provided the webrtc-star module seems to be working correctly with large files. So, my proposal is to try this out with a js-ipfs daemon running instead of the go-ipfs + two browser nodes.

If the problem persists, it will probably be easier to debug. If not, the problem is in the connection with a go node.

[dirkmc]

@jacobheun is it possible this is being caused because js-mplex does not break up a large message into smaller chunks? (as you mentioned in libp2p/specs#118 (comment))

[jacobheun]

I don't think so, the 1MB mplex limit shouldn't cause an issue since it's significantly larger than the 64kb wrtc limit.

If chunking is working fine for direct wrtc connections perhaps there's an issue changing transports (reading over the websocket connection and writing over webrtc). I'd assume this would be fine, but maybe there's an edge case there that bitswap is exposing.

[wehriam]

This application does use PubSub extensively, although the messages should be individually much smaller than 65536 bytes. Are PubSub messages batched at some point and sent without chunking?

[wehriam]

Still seeing this issue in 36.3.

bitswap:network:QmSehzzL:error Error: TypeError: Message too large (can send a maximum of 65536 bytes)
at makeError (https://unpkg.com/[email protected]/dist/index.js:164015:13)
at Peer._write (https://unpkg.com/[email protected]/dist/index.js:163499:27)
at doWrite (https://unpkg.com/[email protected]/dist/index.js:95215:64)
at writeOrBuffer (https://unpkg.com/[email protected]/dist/index.js:95204:5)
at Peer.Writable.write (https://unpkg.com/[email protected]/dist/index.js:95122:11)
at https://unpkg.com/[email protected]/dist/index.js:95764:30
at https://unpkg.com/[email protected]/dist/index.js:33675:9
at http://localhost:3000/cluster/main.js:62267:16
at http://localhost:3000/cluster/main.js:63399:9
at http://localhost:3000/cluster/main.js:63399:9 +2ms

Logging the failed chunk didn't provide much, but might be meaningful to someone in the know:

[jacobheun]

Since the error is coming from bitswap it shouldn't be related to pubsub.
@wehriam have you seen this in any other browsers aside from Chrome? And what version of go-ipfs are you running when hitting this?

My assumption is that when bitswap starts to sharing blocks simple-peer is buffering up the 64k max, and then something else is getting tacked on before sending which would cause RTC to throw the error. Or there is a bug causing the simple-peer buffering and back-pressure to fail.

I am hoping to start spending more time on the webrtc setup over the next couple of weeks and get it added to js-ipfs as a default transport, but I may not get to this bug until the latter part of that time frame.

[wehriam]

go-ipfs version 0.4.20. I've only seen it in Chrome up to this point.

It seems to be intermittent, then happen 10+ times in short succession.

I am hoping to start spending more time on the webrtc setup over the next couple of weeks and get it added to js-ipfs as a default transport, but I may not get to this bug until the latter part of that time frame.

Great news, and apologies if I come across as persistent. JS-IPFS is a tremendous project and I'm anxious to see and use it at it's full potential.

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).

This issue is most likely resolved in the latest Helia version, please try it out!