Peers getting disconnected. Zero active peers.

[niharrs]

💎 Issue

Background

I am setting up an eth2 validator client on testnet. After sometime into the syncing process, peers dropped to zero and I am getting the following log:

Terminal log

[2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101496 already processed [2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101497 already processed [2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101499 already processed [2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101500 already processed [2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101501 already processed [2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101502 already processed [2020-07-28 13:46:46] INFO initial-sync: Block is not processed error=slot 101503 already processed [2020-07-28 13:46:46] ERROR initial-sync: Failed to close stream with protocol /eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy error=read data when expecting EOF [2020-07-28 13:46:48] INFO p2p: Peer disconnected activePeers=13 multiAddr=/ip4/98.254.225.172/tcp/13000/p2p/16Uiu2HAmAq7Jf5x5EJEYjr8JveFoSUT3SEoeE31Guneff1ohamBo [2020-07-28 13:46:52] INFO initial-sync: Processing block 0x0eb50dde... 101536/317697 - estimated time remaining 8h45m56s blocksPerSecond=6.8 peers=13 [2020-07-28 13:47:16] INFO initial-sync: Processing block 0xb1c85825... 101568/317699 - estimated time remaining 10h26m28s blocksPerSecond=5.8 peers=13 [2020-07-28 13:47:19] INFO p2p: Peer disconnected activePeers=12 multiAddr=/ip4/65.32.73.198/tcp/13000/p2p/16Uiu2HAmLwL9UKmpZDSRoaDXNM9aFweetyU7S6HY28hK8ZELXo5L [2020-07-28 13:47:20] INFO initial-sync: Processing block 0x8e02e0fd... 101600/317699 - estimated time remaining 11h53m11s blocksPerSecond=5.0 peers=12 [2020-07-28 13:47:20] INFO p2p: Peer disconnected activePeers=11 multiAddr=/ip4/97.113.158.224/tcp/13000/p2p/16Uiu2HAky3NuR5fkYutAtAANEw2sEzRSkvqoYdHpP1dcyhgvYz6r [2020-07-28 13:47:21] INFO p2p: Peer disconnected activePeers=10 multiAddr=/ip4/95.111.235.154/tcp/13000/p2p/16Uiu2HAmK7ZfWSVAWADjcbugFhGq3R7sP6NZwnLi17LUgziKZtKg [2020-07-28 13:47:21] INFO p2p: Peer disconnected activePeers=9 multiAddr=/ip4/72.192.27.21/tcp/13000/p2p/16Uiu2HAmJY52Yxcy6nvxHSrQyTvVuQ9v3rcwTxgkgjizMRFHyZKd [2020-07-28 13:47:27] ERROR initial-sync: Failed to close stream with protocol /eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy error=read data when expecting EOF [2020-07-28 13:47:27] INFO p2p: Peer disconnected activePeers=8 multiAddr=/ip4/138.88.32.32/tcp/13000/p2p/16Uiu2HAm7ayz3YoACekcDYr5fgfdyNFCMRNXZKb6eVWjEnc627qK [2020-07-28 13:55:01] INFO p2p: Peer disconnected activePeers=7 multiAddr=/ip4/80.144.91.124/tcp/13000/p2p/16Uiu2HAmFrwNQEeiDZ25Dzs9giVYFcHiEk9cUt1k29gRSdACuBSN [2020-07-28 13:55:02] INFO p2p: Peer disconnected activePeers=6 multiAddr=/ip4/173.249.17.215/tcp/13000/p2p/16Uiu2HAkzd3j5dVJcAQ61wUsyicMsSYqHtn83byBuD3jaFAdX6f8 [2020-07-28 13:55:02] INFO p2p: Peer disconnected activePeers=5 multiAddr=/ip4/79.248.105.99/tcp/13000/p2p/16Uiu2HAmEudvqXqvXcmS59jgF1nPmefEBhX9ydaxKCPUsUchpKLG [2020-07-28 13:55:03] INFO p2p: Peer disconnected activePeers=4 multiAddr=/ip4/14.161.38.107/tcp/13000/p2p/16Uiu2HAmJRwaLQSLKtsm5vSHAc9kDoUKwBtANc18eHPxX6Z3CD1u [2020-07-28 13:55:25] ERROR initial-sync: Failed to close stream with protocol /eth2/beacon_chain/req/beacon_blocks_by_range/1/ssz_snappy error=i/o deadline reached [2020-07-28 13:55:27] INFO p2p: Peer connected activePeers=5 direction=Outbound multiAddr=/ip4/173.249.17.215/tcp/13000/p2p/16Uiu2HAkzd3j5dVJcAQ61wUsyicMsSYqHtn83byBuD3jaFAdX6f8 [2020-07-28 13:55:27] INFO initial-sync: Processing block 0x71637c9d... 101632/317740 - estimated time remaining 1200h36m0s blocksPerSecond=0.1 peers=5 [2020-07-28 13:55:29] INFO initial-sync: Processing block 0x73c7494a... 101664/317740 - estimated time remaining 41h23m37s blocksPerSecond=1.4 peers=5 [2020-07-28 13:55:30] INFO initial-sync: Processing block 0x16da6435... 101696/317740 - estimated time remaining 21h49m21s blocksPerSecond=2.8 peers=5 [2020-07-28 13:55:32] INFO p2p: Peer disconnected activePeers=4 multiAddr=/ip4/69.178.221.103/tcp/13000/p2p/16Uiu2HAkyh8w9oVLC2wozYnn6UatNCDqDt65kDLSAqeW9QD5yCiB [2020-07-28 13:55:34] INFO initial-sync: Processing block 0x5e523ca9... 101728/317740 - estimated time remaining 15h23m7s blocksPerSecond=3.9 peers=4 [2020-07-28 14:02:34] INFO p2p: Peer disconnected activePeers=3 multiAddr=/ip4/121.169.133.103/tcp/13000/p2p/16Uiu2HAmEVvBYMJuW98N13iKJy983AYhg3Zavwxz8AVcUzjc3hTZ [2020-07-28 14:02:34] INFO p2p: Peer disconnected activePeers=2 multiAddr=/ip4/54.193.74.5/tcp/13000/p2p/16Uiu2HAmQCZYzbZvgXJR7Gp9Ean1BovYuPQhaRTqBb5ezSUrzMwB [2020-07-28 14:02:34] INFO p2p: Peer disconnected activePeers=1 multiAddr=/ip4/51.81.245.122/tcp/13000/p2p/16Uiu2HAm3PK6dTy1qEdEvAPRHFVjN5zShhb7RqzpqVGeQtzk2nLm [2020-07-28 14:02:35] INFO initial-sync: Processing block 0xd3f30187... 101760/317775 - estimated time remaining 11h25m45s blocksPerSecond=5.2 peers=1 [2020-07-28 14:02:36] INFO initial-sync: Processing block 0x243c1d80... 101792/317776 - estimated time remaining 9h13m48s blocksPerSecond=6.5 peers=1 [2020-07-28 14:02:40] INFO initial-sync: Processing block 0x0149e268... 101856/317776 - estimated time remaining 6h48m56s blocksPerSecond=8.8 peers=1 [2020-07-28 14:02:41] INFO initial-sync: Processing block 0x3102c376... 101888/317776 - estimated time remaining 7h3m18s blocksPerSecond=8.5 peers=1 [2020-07-28 14:02:44] INFO initial-sync: Processing block 0x945df975... 101920/317776 - estimated time remaining 7h8m17s blocksPerSecond=8.4 peers=1 [2020-07-28 14:02:46] INFO p2p: Peer connected activePeers=2 direction=Outbound multiAddr=/ip4/14.161.38.107/tcp/13000/p2p/16Uiu2HAmJRwaLQSLKtsm5vSHAc9kDoUKwBtANc18eHPxX6Z3CD1u [2020-07-28 14:02:47] INFO p2p: Peer disconnected activePeers=1 multiAddr=/ip4/173.249.17.215/tcp/13000/p2p/16Uiu2HAkzd3j5dVJcAQ61wUsyicMsSYqHtn83byBuD3jaFAdX6f8 [2020-07-28 14:02:50] INFO initial-sync: Processing block 0xa249668e... 101952/317777 - estimated time remaining 7h0m42s blocksPerSecond=8.6 peers=1 [2020-07-28 14:05:27] INFO initial-sync: Processing block 0x95a53052... 101984/317790 - estimated time remaining 6h33m5s blocksPerSecond=9.2 peers=1 [2020-07-28 14:05:29] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=1 [2020-07-28 14:05:29] INFO initial-sync: Processing block 0x60fec478... 102016/317790 - estimated time remaining 6h35m11s blocksPerSecond=9.1 peers=1 [2020-07-28 14:05:29] INFO p2p: Peer disconnected activePeers=0 multiAddr=/ip4/14.161.38.107/tcp/13000/p2p/16Uiu2HAmJRwaLQSLKtsm5vSHAc9kDoUKwBtANc18eHPxX6Z3CD1u [2020-07-28 14:05:33] INFO initial-sync: Processing block 0xb1b6acd2... 102080/317790 - estimated time remaining 6h48m32s blocksPerSecond=8.8 peers=0 [2020-07-28 14:05:34] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:05:35] INFO initial-sync: Processing block 0x8c96448c... 102112/317790 - estimated time remaining 6h46m10s blocksPerSecond=8.8 peers=0 [2020-07-28 14:05:39] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:05:44] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:05:49] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:05:54] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:02] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:07] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:12] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:17] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:22] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:48] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:53] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:06:58] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:07:03] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:09:07] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:09:12] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:09:17] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:09:22] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:26] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:31] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:36] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:41] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:46] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:51] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:10:56] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:01] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:06] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:11] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:16] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:21] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:26] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:31] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:36] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:41] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:46] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:51] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:11:56] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:01] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:06] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:11] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:16] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:21] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:26] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:31] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:36] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:41] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:46] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:51] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:12:56] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:01] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:06] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:11] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:16] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:21] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:26] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:31] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:36] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:41] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:46] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:51] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:13:56] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:14:01] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:14:06] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0 [2020-07-28 14:14:11] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0

[nisdas]

Could you provide more details on the system you were running. Ex: OS, Arch, RAM, etc.
Also was there any trigger for the sudden disconnects or was this something that happened randomly ?

[shayzluf]

not sure if this is related to the issue @farazdagi and myself had the day after launch. we both use mac

[rauljordan]

Please see #6779. If this issue still persists, let us know and we will reopen. There have been significant improvements over the last 2 weeks that prevent these scenarios

[nisdas]

Reopening this as this has been reported again.

[shayzluf]

@farazdagi is it related to peer scoring?

[farazdagi]

Shay, no it is not. As it is currently implemented, peers that are considered by scorer as bad/slow are not banned or removed, their chance to participate in fetching the next batch are diminished (yet, they can still continue participating).

[rauljordan]

No longer been reported and no diagnosis info available to resolve at this point

[tanliwei]

Personal experience on linux for mainnet:
Always show the below info and never syncing data after I decrease the number of p2p-max-peers of prysm.sh beaconchain to 50:
Waiting for enough suitable peers before syncing

Begin to sync data after I increase the number of p2p-max-peers to 75.

[web3d3v]

Much like @tanliwei, I'm on arch linux, latest prysm, latest geth.

Managed to sync up to latest about a week ago, afterwards gracefully shutdown. Since then for a week:

Waiting for enough suitable peers before syncing required=3 suitable=0

Increasing --p2p-max-peers did not solve the issue. Deleting .eth2 and trying to start over did not solve the issue either.

[nisdas]

@web3d3v Can you try running v3.1.1 for prysm and see if that fixes your issue ?

[TifferPelode]

I got the same issue, as I run with command ./prysm.sh beacon-chain --execution-endpoint=http://localhost:8551 --jwt-secret=jwt.hex --checkpoint-sync-url=https://sync.invis.tools --genesis-beacon-api-url=https://sync.invis.tools
Should I add or delete some flags?

[TifferPelode]

I solved this issue by setting port, after I open port 13000/TCP and 12000/UDP, it works.
https://docs.prylabs.network/docs/prysm-usage/p2p-host-ip
I think these should be placed in a more prominent place.

[prestonvanloon]

@mick-prysm do you think we can improve our docs here?

[TangMonk]

I solved this issue by setting port, after I open port 13000/TCP and 12000/UDP, it works. https://docs.prylabs.network/docs/prysm-usage/p2p-host-ip I think these should be placed in a more prominent place.

You are right man, this is the CORRECT answer!
Default Allow inbound and outbound port in conflict with Erigon(self embed beacon client), I am Runing Geth and Erigon both