Need help to solve the synchronization problem.

[okmrs]

Hello there.
I need help. It seems to be my node's problem; geth client is not wrong.

I newly started runnig node, using following client.

• Geth: v1.10.23
• prysm: v3.0.0

I faced problem: the synchronization is going wrong due to BAD BLOCK.
Is there any solution here?

Logs:

2022-08-25T12:20:54.017+09:00	INFO [08-25|03:20:54.016] Skip duplicated bad block number=15,400,100 hash=1e7826..f3db8f
2022-08-25T12:20:54.020+09:00	ERROR[08-25|03:20:54.020]
2022-08-25T12:20:54.020+09:00	########## BAD BLOCK #########
2022-08-25T12:20:54.020+09:00	Chain config: Chain ID: 1 (mainnet)
2022-08-25T12:20:54.020+09:00	Consensus: Beacon (proof-of-stake), merging from Ethash (proof-of-work)
2022-08-25T12:20:54.020+09:00	Pre-Merge hard forks:
2022-08-25T12:20:54.020+09:00	- Homestead: 1150000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
2022-08-25T12:20:54.020+09:00	- DAO Fork: 1920000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
2022-08-25T12:20:54.020+09:00	- Tangerine Whistle (EIP 150): 2463000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
2022-08-25T12:20:54.020+09:00	- Spurious Dragon/1 (EIP 155): 2675000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
2022-08-25T12:20:54.020+09:00	- Byzantium: 4370000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
2022-08-25T12:20:54.020+09:00	- Constantinople: 7280000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
2022-08-25T12:20:54.020+09:00	- Petersburg: 7280000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
2022-08-25T12:20:54.020+09:00	- Istanbul: 9069000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
2022-08-25T12:20:54.020+09:00	- Muir Glacier: 9200000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
2022-08-25T12:20:54.020+09:00	- Berlin: 12244000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
2022-08-25T12:20:54.020+09:00	- London: 12965000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
2022-08-25T12:20:54.020+09:00	- Arrow Glacier: 13773000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
2022-08-25T12:20:54.020+09:00	- Gray Glacier: 15050000 (https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
2022-08-25T12:20:54.020+09:00	Merge configured:
2022-08-25T12:20:54.020+09:00	- Hard-fork specification: https:/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
2022-08-25T12:20:54.020+09:00	- Network known to be merged: false
2022-08-25T12:20:54.020+09:00	- Total terminal difficulty: 58750000000000000000000
2022-08-25T12:20:54.020+09:00	- Merge netsplit block: <nil>
2022-08-25T12:20:54.020+09:00	Number: 15400100
2022-08-25T12:20:54.020+09:00	Hash: 0x1e7826b14e6b307e4ceebe49b7b6f7d62ed724e8cd52725311699bf0cdf3db8f
2022-08-25T12:20:54.020+09:00	0: cumulative: 113324 gas: 113324 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x96bb466e13af49eec9b17f2e3824ff170c912253810be78b397a1661c5a49c1d logs: [0xc002666580 0xc0026666e0 0xc002666840] bloom: 00000000000000000000000000000000000000000000000002000000000000000005400000000000000000000000000002000000080024000000000000000000000000000000000800000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000800002000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000040000000000000002000000000000000000000000000000000000000000010000000000000001200000000010000000000000000000000000000000000000000000000000 state:
2022-08-25T12:20:54.020+09:00	1: cumulative: 312645 gas: 199321 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x2427eba8b61e429fb673be33e13b926ef1a1863bf4c2334c0f16a5de32e00470 logs: [0xc002666f20 0xc002667080 0xc0026671e0 0xc002667340 0xc0026674a0 0xc002667a20 0xc0026b4000 0xc0026b4160 0xc0026b48f0 0xc0026b4bb0 0xc0026b4d10] bloom: 10200000000000000000000080000000000000000000000000000010000000040000000000000000101000001000000002010000080000000000000000000000000000000000000000080808000000202200000000000000000000008000000000000000000000000000000080000000000000400000000000000010000000000000000000000000000000000000800000000000000000080000004000400000000000000100008000000000000000000000000000000000000000002000000000000802000000000000000000020000000100800000001000002000000208000040200000000000000000000000000000000001000000000000020020000000 state:
2022-08-25T12:20:54.021+09:00	2: cumulative: 507488 gas: 194843 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x6738347658c8ff6bfc38cee3ac2f7dddd7e8a8282365e6369a267e277a21086d logs: [0xc00276d6b0 0xc00276d970 0xc00276db80 0xc00276dd90 0xc0027da0b0 0xc0027da2c0 0xc0027da8f0 0xc0027daa50 0xc0027dabb0 0xc0027db290] bloom: 00200000000000000000000080000100000000000000000000000000000000040000000000000080000000000000000006400000080020001000000004000000000000800000000880000008000000200200000080000000000000040000000000040000000000000000000000000000000000000000000000000010000800000000000000000000400000000000000001000000000000080000204000000000080000000100000000000000000100080000000000800000000000200000000000000002000000000000000000804000000004000000001000000000000008000000200000000000000020000000000000400000000000005000000000000000 state:
2022-08-25T12:20:54.021+09:00	3: cumulative: 652396 gas: 144908 contract: 0x0000000000000000000000000000000000000000 status: 0 tx: 0x99f4033029702043ef844e89d88aa774ae2921320bd4c6b8a93f23d642ace700 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
2022-08-25T12:20:54.021+09:00	4: cumulative: 875480 gas: 223084 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc27c012c85dabf79d7d58dec05048bab86fd0a13aed865db15fad2a7b2fcd895 logs: [0xc0027f6210 0xc0027f6370 0xc0021ad290 0xc0021ad3f0 0xc0021ad550 0xc0021ad6b0 0xc0021ad810] bloom: 00040020010000000000000000000004000000010000000000000000040000000000000000000000001008000000000002000000080020000000000000000000000080000000000808000208000000002000000000001000000000000000000000000000000000000002000000000000000000000000000001000010000800000000000000000000000000000000000000000000010000000000000000000400000000000040200800000000000000000000000000000000002000000008000000000002000000000000000000000000000000000000000000400000000000000000200000000000000010000000000000000000004010000000000000000000 state:
2022-08-25T12:20:54.021+09:00	5: cumulative: 1056045 gas: 180565 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc2b8c1a588e3fe6c527cc6134e6d21591f9c4e55ab3fe938df47c48b62afdb4f logs: [0xc0021ad970 0xc0021adad0 0xc0021adc30 0xc0021add90 0xc0021adef0] bloom: 00000000000800000000000000000400000000000000000000000000000000000000000008000000000000000000000002000000080020010000000000000000000000000008000800000008008000000000000000000000000000000000400000000000000004000000000000000000000000000000000100000014000800000000000000000000000000000000000000400000000000000000000000004000000000000040000004000000000000000000000000000000000008000000000000000002000000000000400000000000000000000000000000000000000000000000200000000000000000000000000000000000000000400000000000020000 state:
(...)
2022-08-25T12:20:54.022+09:00	90: cumulative: 15782341 gas: 46483 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x7aa1b699cf8e9704f32ae9126613a2079c16d7fa0b19982895769b4aec1c55da logs: [0xc00394c580] bloom: 00000000000000800000004000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000000000100000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000010000001000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001000000000000000000010000000000000000000000000000000000000000000010002000000000000000000000000000000000000000000000000000000000 state:
2022-08-25T12:20:54.022+09:00	91: cumulative: 16012642 gas: 230301 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xaad25e60acf81dd44ea00623090db61f916383537b2ca4c1ea7c29375b9ddc27 logs: [0xc00394c840 0xc00394cb00 0xc00394cdc0 0xc00394d080 0xc00394d3f0 0xc00394d6b0 0xc0039606e0] bloom: 00200000000400000000000080000000000000000000000000000000000000000000000000000000000000000000001002000000080000000000000001000000000000000042000040000008100000200200000000008000000000008000000008000000000000001000000000000000000100000000000000000210800000001000000000000040000800000000000000000001000000080000004000000000000000000000000000000000000000000000000400000000000000000000000000000002000000000000000000000000000000000010001000000100000000000000200000000000000000000000080000000000001000400000000000000000 state:
2022-08-25T12:20:54.022+09:00	92: cumulative: 16058538 gas: 45896 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x44a3fb0c89be49df7c9ef30813cb7e8849d22c397262d69381c5119329f9c5a6 logs: [0xc003960b00] bloom: 00000000000000000000000000100000000000000000000000000000000000000000000000000000000000001000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000010000000000008000000000000004000000000000000000000000000000000000000000040000800000000000000000000000000000000000040000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000 state:
2022-08-25T12:20:54.022+09:00	93: cumulative: 16112578 gas: 54040 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xaf073c465ca93f492f4ada73141fc3039b9a2197591eaaae5dd82c382e380ce8 logs: [0xc003960fd0] bloom: 00000000000000000000000800000200000000000000000000000000000000000000000000000000000000100000000000000000000000000200000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000002000000000000000000000000000800000000000000000000000000000000000000000000000002000010000000000000000000000000000000000000 state:
2022-08-25T12:20:54.022+09:00	94: cumulative: 16161103 gas: 48525 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x274d2eff9cc0984128a417f5025dfc6d8cbde5fc11933ba7ea93147f5c6dcdaa logs: [0xc003961130] bloom: 00000000000400000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000080000000000000000000000008000008000000000000000000400000000800000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000010000000000000000000000400000000000200000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
2022-08-25T12:20:54.022+09:00	Error: invalid gas used (remote: 16230876 local: 16161103)
2022-08-25T12:20:54.022+09:00	##############################
2022-08-25T12:20:54.022+09:00	ERROR[08-25|03:20:54.020] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"

[SjonHortensius]

is it possible the ancient directory got corrupted by the 1.10.22 release? I have the same error, trying prune-state throws snapshot not old enough yet: need 74 more blocks

[karalabe]

The chaindata was corrupted in 1.10.22.

Honestly the simplest solution is to do geth removedb, deleting the chaindata but keeping the ancients and resync. It's sure and hopefully doesn't take too much.

[SjonHortensius]

I experienced this issue with 1.10.23 with the chaindata removed (but ancient intact)

...
INFO [08-24|14:32:30.781] Iterating state snapshot                 accounts=180,251,957 slots=768,711,192 elapsed=47m12.039s eta=3.738s
INFO [08-24|14:32:30.864] Iterated snapshot                        accounts=180,251,957 slots=769,821,685 elapsed=47m12.122s
INFO [08-24|14:32:42.883] Writing state bloom to disk              name=/var/lib/geth/geth/statebloom.0x9ec800471f014c26c91b4df66d487c6b0795c17d03655f284eb582b3c21a3d3b.bf.gz
INFO [08-24|14:33:32.582] State bloom filter committed             name=/var/lib/geth/geth/statebloom.0x9ec800471f014c26c91b4df66d487c6b0795c17d03655f284eb582b3c21a3d3b.bf.gz
INFO [08-24|14:33:40.582] Pruning state data                       nodes=185,593 size=60.68MiB elapsed=8.000s     eta=10m42.048s
...
INFO [08-24|14:54:01.350] Pruning state data                       nodes=15,012,925 size=4.80GiB    elapsed=20m28.768s eta=6.269s
INFO [08-24|14:54:06.303] Pruned state data                        nodes=15,089,582 size=4.82GiB    elapsed=20m33.721s
INFO [08-24|14:54:06.366] Compacting database                      range=0x00-0x10 elapsed="2.375µs"
...
INFO [08-24|16:42:52.651] Database compaction finished             elapsed=1h48m46.285s
INFO [08-24|16:42:52.651] State pruning successful                 pruned=4.82GiB elapsed=2h57m33.911s
INFO [08-24|16:42:53.022] Starting Geth on Ethereum mainnet...
INFO [08-24|16:42:53.024] Maximum peer count                       ETH=50 LES=0 total=50
INFO [08-24|16:42:53.025] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [08-24|16:42:53.027] Set global gas cap                       cap=50,000,000
INFO [08-24|16:42:53.029] Allocated trie memory caches             clean=153.00MiB dirty=256.00MiB
INFO [08-24|16:42:53.029] Allocated cache and file handles         database=/var/lib/geth/geth/chaindata cache=512.00MiB handles=10240
INFO [08-24|16:42:53.983] Found legacy ancient chain path          location=/var/lib/geth/geth/chaindata/ancient
INFO [08-24|16:42:58.828] Opened ancient database                  database=/var/lib/geth/geth/chaindata/ancient readonly=false
...
INFO [08-24|16:42:59.939] Loaded most recent local header          number=15,402,738 hash=7a2481..ca8ef6 td=57,095,651,013,178,184,317,918 age=2h58m15s
INFO [08-24|16:42:59.939] Loaded most recent local full block      number=15,402,738 hash=7a2481..ca8ef6 td=57,095,651,013,178,184,317,918 age=2h58m15s
INFO [08-24|16:42:59.939] Loaded most recent local fast block      number=15,402,738 hash=7a2481..ca8ef6 td=57,095,651,013,178,184,317,918 age=2h58m15s
INFO [08-24|16:42:59.939] Loaded last fast-sync pivot marker       number=15,393,404
WARN [08-24|16:42:59.940] Head state missing, repairing            number=15,402,738 hash=7a2481..ca8ef6 snaproot=9ec800..1a3d3b
INFO [08-24|16:43:00.065] Loaded most recent local header          number=15,402,738 hash=7a2481..ca8ef6 td=57,095,651,013,178,184,317,918 age=2h58m16s
INFO [08-24|16:43:00.065] Loaded most recent local full block      number=15,402,611 hash=364b0c..843505 td=57,094,095,423,935,344,761,779 age=3h23m49s
INFO [08-24|16:43:00.065] Loaded most recent local fast block      number=15,402,738 hash=7a2481..ca8ef6 td=57,095,651,013,178,184,317,918 age=2h58m16s
INFO [08-24|16:43:00.065] Loaded last fast-sync pivot marker       number=15,393,404
INFO [08-24|16:43:00.067] Loaded local transaction journal         transactions=0 dropped=0
INFO [08-24|16:43:00.067] Regenerated local transaction journal    transactions=0 accounts=0
WARN [08-24|16:43:00.067] Switch sync mode from snap sync to full sync
WARN [08-24|16:43:00.067] Chain pre-merge, sync via PoW (ensure beacon client is ready)
INFO [08-24|16:43:00.067] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-24|16:43:00.068] Engine API enabled                       protocol=eth
INFO [08-24|16:43:00.118] Starting peer-to-peer node               instance=Geth/v1.10.23-stable-d901d853/linux-amd64/go1.19
...
INFO [08-24|16:43:00.293] Loaded JWT secret file                   path=/var/lib/geth/geth/jwtsecret crc32=0xe99eee1a
INFO [08-24|16:43:00.293] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors= vhosts=*
INFO [08-24|16:43:00.294] WebSocket enabled                        url=ws://[::]:8551
INFO [08-24|16:43:00.294] HTTP server started                      endpoint=[::]:8551 auth=true  prefix= cors=localhost vhosts=*
INFO [08-24|16:43:06.900] Block synchronisation started
WARN [08-24|16:43:06.995] Synchronisation failed, dropping peer    peer=e00ec0f119f846d9cae8e592dc75b6ae9b20555c4b99dc18ef5ce00488160b62 err="action from bad peer ignored: returned >
INFO [08-24|16:43:07.194] New local node record                    ...
WARN [08-24|16:43:07.245] Synchronisation failed, dropping peer    peer=9c847a3d60a7c8a480524d97a47c1914384799e205c2a294cbd74ac51a8eece5 err="action from bad peer ignored: returned >
INFO [08-24|16:43:10.521] Looking for peers                        peercount=4 tried=15 static=0
INFO [08-24|16:43:12.841] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=57.30B throttle=8192
INFO [08-24|16:43:12.950] Importing sidechain segment              start=15,402,612 end=15,402,738
WARN [08-24|16:43:19.247] Snapshot extension registration failed   peer=4dfa8e8a err="peer connected on snap without compatible eth support"
INFO [08-24|16:43:20.854] Looking for peers                        peercount=12 tried=28 static=0
INFO [08-24|16:43:20.962] Imported new chain segment               blocks=4 txs=1012 mgas=82.344 elapsed=8.011s mgasps=10.279 number=15,402,615 hash=8c5d3a..c39917 age=3h22m27s dirt>
INFO [08-24|16:43:29.256] Imported new chain segment               blocks=15 txs=2153 mgas=186.611 elapsed=8.294s mgasps=22.500 number=15,402,630 hash=a4721d..3a0346 age=3h20m8s  di>
INFO [08-24|16:43:30.855] Looking for peers                        peercount=16 tried=26 static=0
INFO [08-24|16:43:37.801] Imported new chain segment               blocks=28 txs=4093 mgas=375.122 elapsed=8.545s mgasps=43.897 number=15,402,658 hash=1ba0df..834401 age=3h16m4s  di>
WARN [08-24|16:43:40.556] Trie prefetcher failed opening trie      root=461acf..cf162d err="missing trie node 461acf044e171145682d0b5b934246df69094def36cf79b278dba10005cf162d (owner>
WARN [08-24|16:43:40.926] Failed to load old bad blocks            error="leveldb: not found"
ERROR[08-24|16:43:41.032]
########## BAD BLOCK #########

[okmrs]

I couldn't execute geth removedb

INFO [08-25|11:08:44.735] Maximum peer count                       ETH=50 LES=0 total=50
INFO [08-25|11:08:44.743] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Fatal: Failed to create the protocol stack: datadir already used by another process

so tried another way like this: #25589 (comment)

Node restarted synchronizing.
I'll wait and see...

[holiman]

Fatal: Failed to create the protocol stack: datadir already used by another process

You cannot do geth removedb while geth is running

[holiman]

@SjonHortensius

with the chaindata removed (but ancient intact)

INFO [08-24|16:42:59.939] Loaded most recent local header number=15,402,738

What do you mean? That log is from a node which most definitely has chaindata intact, not removed at all

[geomad]

@SjonHortensius

with the chaindata removed (but ancient intact)

INFO [08-24|16:42:59.939] Loaded most recent local header number=15,402,738

What do you mean? That log is from a node which most definitely has chaindata intact, not removed at all

I also removedb , and was getting a similar message...
I ended up to removedb again, and went ahead and deleted everything inside the geth folder except ancients.
Logs show this now

Loaded most recent local header          number=15,318,558 hash=5fc06c..b8567c td=56,087,366,332,840,166,971,645 age=2w8h15m
Loaded most recent local full block      number=0          hash=d4e567..cb8fa3 td=17,179,869,184                 age=53y4mo4w
Loaded most recent local fast block      number=15,318,558 hash=5fc06c..b8567c td=56,087,366,332,840,166,971,645 age=2w8h15m

[SjonHortensius]

@SjonHortensius

with the chaindata removed (but ancient intact)

INFO [08-24|16:42:59.939] Loaded most recent local header number=15,402,738

What do you mean? That log is from a node which most definitely has chaindata intact, not removed at all

I'm not sure what happened, but doesn't the State pruning successful message confirm the chaindata was cleared?

[holiman]

I'm not sure what happened, but doesn't the State pruning successful

No, not at all. State pruning is something entirely different. To remove the chain data, but leave ancients intact, use geth removedb, and answer accordingly.

[SjonHortensius]

I'm not sure what happened, but doesn't the State pruning successful

No, not at all. State pruning is something entirely different. To remove the chain data, but leave ancients intact, use geth removedb, and answer accordingly.

oops, you're right of course. Sorry for the confusion

[okmrs]

Finally, completed sync!!
Thank you for your cooperation.

Close.