Forest can't bootstrap a Lotus node

[LesnyRumcajs]

Issue summary

It turns out a Forest node (v0.19.0) is not a able to bootstrap a Lotus node. It can, however, bootstrap another Forest node.

To reproduce on calibnet:

1. Start a Forest node, grab it's multiaddr with peer id.
2. Compile Lotus with the above as bootstrap node (make sure to set LOTUS_SYNC_BOOTSTRAP_PEERS=1)
3. With lotus net peers observe that the only peer it is connected to is the bootstrap node, whereas forest-cli net peers shows way more peers.

Similarly, do the same with another Forest and observe that the second instance has many other peers.

This behaviour might stem from the fact that the peer exchange is not implemented in gossipsub in rust-libp2p ( code mention and issue) while explicitly enabled in Lotus.

Does peer exchange over Kademlia and Identify not work in Lotus? If so, can this be mitigated? The alternative would be implementing peer exchange in gossipsub, which to my understanding, is not a trivial task.

To work on this, and ensure it doesn't break in the future, once fixed, we also need a CI job to check Forest-bootstrapper compatibility with Lotus (and another Forest).

Other information and links

[LesnyRumcajs]

@hanabi1224 I'll work on CI job to check Forest-bootstrapper compatibility with Lotus.

[hanabi1224]

The issue has been 'fixed' or mitigated by #4580 as a workaround
We decided to keep the issue open until a proper fix is made to the upstream rust-libp2p, go-libp2p-kad-dht or lotus

[hanabi1224]

Confirmed that the issue has been fixed in rust-libp2p, see libp2p/rust-libp2p#5269
We just need to upgrade libp2p-kad once it's released and revert #4580

[LesnyRumcajs]

Forest can bootstrap a Lotus node now, so closing. We'll track the removal of workaround in #4602