Frequent invalid merkle state root BAD BLOCK errors on all nodes

[tnunamak]

System information

Geth version: 1.13.15-stable
CL client & version: none, using Clique PoA
OS & Version: Linux
Commit hash: c5ba367

Expected behaviour

Applying proposed blocks containing transactions should result in the same state root across synchronized validators.

Actual behaviour

A simple token send transaction results in an error: invalid merkle root (remote: 020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618d local: c5efce337aa62af3fe0dca863454b3cd183dd2ea73f0c1e85e2dbf065ac432cb)

Steps to reproduce the behaviour

I have deployed a PoA chain using the last PoA-supported version of geth. The network configuration is three fully-connected mining node and one non-mining RPC node on VMs in Google Cloud, each running geth via docker compose.

Some details:

• This seems to happen occasionally for blocks containing transactions (but even simple transactions like sending ETH) but not for blocks containing no transactions.
• All validators agree on the previous state root.
• As mentioned, I'm using vanilla geth and haven't changed geth's source code (e.g. to modify state transitions).
• When multiple validators try to propose the same block, they calculate the same/correct state root when I check it locally with eth.getBlock(). It's only when validators are processing each other's transactions that the state root calculation seems to be different.
• The chain has ~300k blocks and maybe ~100k transactions since it was spun up a few weeks ago. I'm seeing an increasing number of bad blocks and consensus occasionally fails because all validators calculate incorrect local merkle roots.
• Validators are running on separate machines, in VMs, via docker. All validators have this behavior. No disk corruption was detected when tested.

genesis.json

  {
    "config": {
      "chainId": 14801,
      "homesteadBlock": 0,
      "eip150Block": 0,
      "eip155Block": 0,
      "eip158Block": 0,
      "byzantiumBlock": 0,
      "constantinopleBlock": 0,
      "petersburgBlock": 0,
      "istanbulBlock": 0,
      "berlinBlock": 0,
      "clique": {
        "period": 6,
        "epoch": 30000
      }
    },
    "coinbase": "0x0000000000000000000000000000000000000000",
    "difficulty": "1",
    "gasLimit": "8000000",
    "extradata": "0x0000000000000000000000000000000000000000000000000000000000000000f94d7de831894759ccf3805eb4b12ec67fd37367a2839e3ae2c2b1c30df129a0558a6a34757d5712943ae740dc9aebd3aebd2fd15f377260e95b84980000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "alloc": { ... }

geth options:

GETH_NETWORKID=14801
GETH_BOOTNODES=...all nodes...
GETH_UNLOCK=$MINER_ADDRESS
GETH_MINER_ETHERBASE=$MINER_ADDRESS
GETH_GCMODE=archive
GETH_MINE=true
GETH_ALLOW_INSECURE_UNLOCK=true
GETH_PASSWORD=/root/.ethereum/password.txt
GETH_HTTP=true
GETH_HTTP_ADDR=0.0.0.0
GETH_HTTP_PORT=80
GETH_HTTP_API=admin,personal,eth,net,web3,txpool,miner,debug
GETH_HTTP_CORSDOMAIN=*
GETH_HTTP_VHOSTS=*
GETH_NODEKEY=/root/boot.key

The full deployment and configuration is public and visible here (satori).

Logs

Validator 1

INFO [06-26|20:46:55.007] Commit new sealing work                  number=361,266 sealhash=c29745..8e1667 txs=0 gas=0     fees=0            elapsed="433.721µs"
WARN [06-26|20:46:55.007] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:00.687] Looking for peers                        peercount=3 tried=183 static=0
INFO [06-26|20:47:01.043] Imported new chain segment               number=361,266 hash=536cc9..916ee2 blocks=1   txs=0 mgas=0.000 elapsed=5.755ms     mgasps=0.000 snapdiffs=128.24KiB triedirty=0.00B
INFO [06-26|20:47:01.043] Commit new sealing work                  number=361,267 sealhash=8f01c0..10c068 txs=0 gas=0     fees=0            elapsed="225.491µs"
INFO [06-26|20:47:03.148] Submitted transaction                    hash=0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c from=0x100485687730c47ba1D40e5f3b143A1492c5F963 nonce=17 recipient=0x100485687730c47ba1D40e5f3b143A1492c5F963 value=0
INFO [06-26|20:47:05.045] Commit new sealing work                  number=361,267 sealhash=47a3dd..8b477f txs=1 gas=21000 fees=2.52e-05     elapsed="399.072µs"
ERROR[06-26|20:47:07.040] 
########## BAD BLOCK #########
Block: 361267 (0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da)
Error: invalid merkle root (remote: 020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618d local: c5efce337aa62af3fe0dca863454b3cd183dd2ea73f0c1e85e2dbf065ac432cb) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.9 amd64 linux
VCS: c5ba367e-
Chain config: &params.ChainConfig{ChainID:14801, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:<nil>, BerlinBlock:0, LondonBlock:<nil>, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(nil), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(0xc000a397f0)}
Receipts: 
  0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
##############################

INFO [06-26|20:47:07.235] Successfully sealed new block            number=361,267 sealhash=47a3dd..8b477f hash=5883f3..70c8db elapsed=2.190s
INFO [06-26|20:47:07.236] Commit new sealing work                  number=361,268 sealhash=337403..974ce3 txs=0 gas=0     fees=0            elapsed="638.052µs"
WARN [06-26|20:47:07.236] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:10.766] Looking for peers                        peercount=3 tried=129 static=0
INFO [06-26|20:47:13.122] Imported new chain segment               number=361,268 hash=85a16a..d97a79 blocks=1   txs=0 mgas=0.000 elapsed=5.932ms     mgasps=0.000 snapdiffs=128.33KiB triedirty=0.00B
INFO [06-26|20:47:13.122] Commit new sealing work                  number=361,269 sealhash=6da600..450b97 txs=0 gas=0     fees=0            elapsed="378.282µs"
INFO [06-26|20:47:19.311] Successfully sealed new block            number=361,269 sealhash=6da600..450b97 hash=7181b2..f75a57 elapsed=6.188s
INFO [06-26|20:47:19.311] Commit new sealing work                  number=361,270 sealhash=b6c787..519120 txs=0 gas=0     fees=0            elapsed="537.949µs"
WARN [06-26|20:47:19.311] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:20.784] Looking for peers                        peercount=3 tried=174 static=0
INFO [06-26|20:47:25.571] Imported new chain segment               number=361,270 hash=8fb0bd..19f1f6 blocks=1   txs=0 mgas=0.000 elapsed=6.036ms     mgasps=0.000 snapdiffs=128.33KiB triedirty=0.00B
INFO [06-26|20:47:25.572] Commit new sealing work                  number=361,271 sealhash=64f9c6..43d140 txs=0 gas=0     fees=0            elapsed="334.435µs"
INFO [06-26|20:47:30.837] Looking for peers                        peercount=3 tried=214 static=0
INFO [06-26|20:47:31.006] Successfully sealed new block            number=361,271 sealhash=64f9c6..43d140 hash=8de1f0..d58567 elapsed=5.434s

Validator 2

INFO [06-26|20:47:01.042] Imported new chain segment               number=361,266 hash=536cc9..916ee2 blocks=1 txs=0 mgas=0.000 elapsed=7.032ms     mgasps=0.000 snapdiffs=128.10KiB triedirty=0.00B
INFO [06-26|20:47:01.043] Commit new sealing work                  number=361,267 sealhash=24687d..63b27c txs=0 gas=0     fees=0            elapsed="588.652µs"
INFO [06-26|20:47:05.045] Commit new sealing work                  number=361,267 sealhash=415859..4fbb3b txs=1 gas=21000 fees=2.52e-05     elapsed="441.508µs"
INFO [06-26|20:47:07.014] Successfully sealed new block            number=361,267 sealhash=415859..4fbb3b hash=cf8c30..f680da elapsed=1.969s
INFO [06-26|20:47:07.015] Commit new sealing work                  number=361,268 sealhash=cb34c9..03b171 txs=0 gas=0     fees=0            elapsed="523.014µs"
WARN [06-26|20:47:07.015] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:07.264] Imported new chain segment               number=361,267 hash=5883f3..70c8db blocks=1 txs=1 mgas=0.021 elapsed=11.769ms    mgasps=1.784 snapdiffs=128.28KiB triedirty=0.00B
INFO [06-26|20:47:13.154] Imported new chain segment               number=361,268 hash=85a16a..d97a79 blocks=1 txs=0 mgas=0.000 elapsed=3.709ms     mgasps=0.000 snapdiffs=128.28KiB triedirty=0.00B
INFO [06-26|20:47:25.072] Chain reorg detected                     number=361,266 hash=536cc9..916ee2 drop=1 dropfrom=cf8c30..f680da add=3 addfrom=7181b2..f75a57
INFO [06-26|20:47:25.165] Imported new chain segment               number=361,270 hash=8fb0bd..19f1f6 blocks=2 txs=0 mgas=0.000 elapsed=98.499ms    mgasps=0.000 snapdiffs=128.28KiB triedirty=0.00B
INFO [06-26|20:47:25.166] Commit new sealing work                  number=361,271 sealhash=ea0a3d..19cbf1 txs=0 gas=0     fees=0            elapsed="344.882µs"

Validator 3

INFO [06-26|20:46:55.017] Commit new sealing work                  number=361,266 sealhash=06b013..b24dc2 txs=0 gas=0     fees=0            elapsed="261.329µs"
INFO [06-26|20:47:01.034] Successfully sealed new block            number=361,266 sealhash=06b013..b24dc2 hash=536cc9..916ee2 elapsed=6.016s
INFO [06-26|20:47:01.035] Commit new sealing work                  number=361,267 sealhash=8f01c0..10c068 txs=0 gas=0     fees=0            elapsed="398.324µs"
WARN [06-26|20:47:01.035] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:03.353] Looking for peers                        peercount=3 tried=244 static=0
INFO [06-26|20:47:05.036] Commit new sealing work                  number=361,267 sealhash=70e772..724a05 txs=1 gas=21000 fees=2.52e-05     elapsed="414.915µs"
WARN [06-26|20:47:05.036] Block sealing failed                     err="signed recently, must wait for others"
ERROR[06-26|20:47:07.019] 
########## BAD BLOCK #########
Block: 361267 (0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da)
Error: invalid merkle root (remote: 020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618d local: c5efce337aa62af3fe0dca863454b3cd183dd2ea73f0c1e85e2dbf065ac432cb) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.9 amd64 linux
VCS: c5ba367e-
Chain config: &params.ChainConfig{ChainID:14801, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:<nil>, BerlinBlock:0, LondonBlock:<nil>, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(nil), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(0xc000464d70)}
Receipts: 
  0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
##############################

INFO [06-26|20:47:07.066] Skip duplicated bad block                number=361,267 hash=cf8c30..f680da
ERROR[06-26|20:47:07.066] 
########## BAD BLOCK #########
Block: 361267 (0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da)
Error: invalid merkle root (remote: 020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618d local: c5efce337aa62af3fe0dca863454b3cd183dd2ea73f0c1e85e2dbf065ac432cb) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.9 amd64 linux
VCS: c5ba367e-
Chain config: &params.ChainConfig{ChainID:14801, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:<nil>, BerlinBlock:0, LondonBlock:<nil>, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(nil), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(0xc000464d70)}
Receipts: 
  0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
##############################

INFO [06-26|20:47:07.266] Imported new chain segment               number=361,267 hash=5883f3..70c8db blocks=1 txs=1 mgas=0.021 elapsed=12.514ms    mgasps=1.678 snapdiffs=512.00B triedirty=0.00B
INFO [06-26|20:47:07.267] Commit new sealing work                  number=361,268 sealhash=03d9d8..d37ec9 txs=0 gas=0     fees=0            elapsed="529.809µs"
INFO [06-26|20:47:13.095] Successfully sealed new block            number=361,268 sealhash=03d9d8..d37ec9 hash=85a16a..d97a79 elapsed=5.828s
INFO [06-26|20:47:13.096] Commit new sealing work                  number=361,269 sealhash=49ea0f..3f5613 txs=0 gas=0     fees=0            elapsed="488.458µs"
WARN [06-26|20:47:13.096] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:13.368] Looking for peers                        peercount=3 tried=279 static=0
INFO [06-26|20:47:19.859] Imported new chain segment               number=361,269 hash=7181b2..f75a57 blocks=1 txs=0 mgas=0.000 elapsed=45.264ms    mgasps=0.000 snapdiffs=512.00B triedirty=0.00B
INFO [06-26|20:47:19.859] Commit new sealing work                  number=361,270 sealhash=b6c787..519120 txs=0 gas=0     fees=0            elapsed="573.175µs"
INFO [06-26|20:47:23.375] Looking for peers                        peercount=3 tried=234 static=0
INFO [06-26|20:47:25.063] Successfully sealed new block            number=361,270 sealhash=b6c787..519120 hash=8fb0bd..19f1f6 elapsed=5.203s
INFO [06-26|20:47:25.063] Commit new sealing work                  number=361,271 sealhash=ea0a3d..19cbf1 txs=0 gas=0     fees=0            elapsed="327.866µs"
WARN [06-26|20:47:25.063] Block sealing failed                     err="signed recently, must wait for others"
INFO [06-26|20:47:31.023] Imported new chain segment               number=361,271 hash=8de1f0..d58567 blocks=1 txs=0 mgas=0.000 elapsed=6.830ms     mgasps=0.000 snapdiffs=512.00B triedirty=0.00B

Console

On Validator 1:

> badBlock = debug.getBadBlocks()[0]
{
  block: {
    difficulty: "0x2",
    extraData: "0xd883010d0f846765746888676f312e32312e39856c696e757800000000000000ccea628bdc5e58edf56b1b334b5aba13f936ac4148d8b97f17076a1fac461fb05932b296c8e5c4ae4c5320fa13aa9ca33e63d4821d91260dfacc2989b37af53800",
    gasLimit: "0x1c9c380",
    gasUsed: "0x5208",
    hash: "0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da",
    logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    miner: "0x0000000000000000000000000000000000000000",
    mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
    nonce: "0x0000000000000000",
    number: "0x58333",
    parentHash: "0x536cc9b31fb9c780297ccd3c62c6aff28fd71f4f02c2312131b3c2a8bd916ee2",
    receiptsRoot: "0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2",
    sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    size: "0x2cc",
    stateRoot: "0x020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618d",
    timestamp: "0x667c7e4b",
    transactions: [{
        blockHash: "0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da",
        blockNumber: "0x58333",
        chainId: "0x39d1",
        from: "0x100485687730c47ba1d40e5f3b143a1492c5f963",
        gas: "0x7b0c",
        gasPrice: "0x47868c00",
        hash: "0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c",
        input: "0x",
        nonce: "0x11",
        r: "0xb056cb542cfba49301d0ad80b5c00415a03decb1adc1b9d946e2a07e8c7e57b",
        s: "0x1ec842c05de90ef52374f91d43b0016a6d2b361255bd2296b16f869e2ab4888b",
        to: "0x100485687730c47ba1d40e5f3b143a1492c5f963",
        transactionIndex: "0x0",
        type: "0x0",
        v: "0x73c6",
        value: "0x0"
    }],
    transactionsRoot: "0x2e4d91197891c085c1a3f1e827fa76f7d41b6e5bd1d9a962d06817a840d51920",
    uncles: []
  },
  hash: "0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da",
  rlp: "0xf902c9f9025ca0536cc9b31fb9c780297ccd3c62c6aff28fd71f4f02c2312131b3c2a8bd916ee2a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347940000000000000000000000000000000000000000a0020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618da02e4d91197891c085c1a3f1e827fa76f7d41b6e5bd1d9a962d06817a840d51920a0056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2b901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002830583338401c9c38082520884667c7e4bb861d883010d0f846765746888676f312e32312e39856c696e757800000000000000ccea628bdc5e58edf56b1b334b5aba13f936ac4148d8b97f17076a1fac461fb05932b296c8e5c4ae4c5320fa13aa9ca33e63d4821d91260dfacc2989b37af53800a00000000000000000000000000000000000000000000000000000000000000000880000000000000000f867f865118447868c00827b0c94100485687730c47ba1d40e5f3b143a1492c5f96380808273c6a00b056cb542cfba49301d0ad80b5c00415a03decb1adc1b9d946e2a07e8c7e57ba01ec842c05de90ef52374f91d43b0016a6d2b361255bd2296b16f869e2ab4888bc0"
}
> goodBlock = eth.getBlock(parseInt(badBlock.number, 16), true);
{
  difficulty: 1,
  extraData: "0xd883010d0f846765746888676f312e32312e39856c696e757800000000000000191b4404a6c029a10989a7570ff7883217d4ebbcb6b3a764f4fd40c045955ccc72df33f9a82de70563b08253dd3dd4d213741418ef41917d8468207c58a8634001",
  gasLimit: 30000000,
  gasUsed: 21000,
  hash: "0x5883f36c90cea8720fb1f400ec30c9e293821a3864d1654e8d2a2a2d0670c8db",
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  miner: "0x0000000000000000000000000000000000000000",
  mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000",
  nonce: "0x0000000000000000",
  number: 361267,
  parentHash: "0x536cc9b31fb9c780297ccd3c62c6aff28fd71f4f02c2312131b3c2a8bd916ee2",
  receiptsRoot: "0x056b23fbba480696b65fe5a59b8f2148a1299103c4f57df839233af2cf4ca2d2",
  sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
  size: 716,
  stateRoot: "0x103480ae2d194df402c4de19076798d96a9bc4ba0603d7b7dd3738fd1c1d4548",
  timestamp: 1719434827,
  totalDifficulty: 708870,
  transactions: [{
      blockHash: "0x5883f36c90cea8720fb1f400ec30c9e293821a3864d1654e8d2a2a2d0670c8db",
      blockNumber: 361267,
      chainId: "0x39d1",
      from: "0x100485687730c47ba1d40e5f3b143a1492c5f963",
      gas: 31500,
      gasPrice: 1200000000,
      hash: "0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c",
      input: "0x",
      nonce: 17,
      r: "0xb056cb542cfba49301d0ad80b5c00415a03decb1adc1b9d946e2a07e8c7e57b",
      s: "0x1ec842c05de90ef52374f91d43b0016a6d2b361255bd2296b16f869e2ab4888b",
      to: "0x100485687730c47ba1d40e5f3b143a1492c5f963",
      transactionIndex: 0,
      type: "0x0",
      v: "0x73c6",
      value: 0
  }],
  transactionsRoot: "0x2e4d91197891c085c1a3f1e827fa76f7d41b6e5bd1d9a962d06817a840d51920",
  uncles: []
}

[rjl493456442]

The block Block: 361267 (0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da) is reported as BAD block.

########## BAD BLOCK #########
Block: 361267 (0xcf8c304e7078d3e3730a87865ade88cc8588d578a02c1090918d34446bf680da)
Error: invalid merkle root (remote: 020c9b1592689f364f784224287e7b7844fffa4efd697798ddce55897fe3618d local: c5efce337aa62af3fe0dca863454b3cd183dd2ea73f0c1e85e2dbf065ac432cb) dberr: %!w(<nil>)
Platform: geth (devel) go1.21.9 amd64 linux
VCS: c5ba367e-
Chain config: &params.ChainConfig{ChainID:14801, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:<nil>, BerlinBlock:0, LondonBlock:<nil>, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(nil), CancunTime:(*uint64)(nil), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:<nil>, TerminalTotalDifficultyPassed:false, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(0xc000a397f0)}
Receipts: 
  0: cumulative: 21000 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xc1a17de0ddc2d98af779fae74578895222c9304c5e1941bb6fbbf57bef10585c logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state: 
##############################

And the correct one is INFO [06-26|20:47:07.266] Imported new chain segment number=361,267 hash=5883f3..70c8db blocks=1 txs=1 mgas=0.021 elapsed=12.514ms mgasps=1.678 snapdiffs=512.00B triedirty=0.00B . Can you double check whether one of your block producer generated this invalid block?

[karalabe]

As long as your chain is progressing, the issue seems that one signer is corrupted and it's trying to make bad block every time it is it's turn. You need to check that machine whether it's SSD is borked or just a db corruption (srrange).