panic: deletion not supported #475

vrogojin · 2021-10-23T10:48:54Z

Started fresh full node with snap sync
geth --syncmode=snap --config ./config.toml --datadir node --cache 18000 --rpc --http.port 9545 --http.corsdomain "*" --http.api "eth,web3,personal,net,txpool" --ws --ws.port 8546 --ws.origins "*" \ --ws.api "eth,web3,personal,net,txpool" --rpc.allow-unprotected-txs --txlookuplimit 0 console

After around an hour it crashes


INFO [10-23|00:29:36.318] Starting Geth on Ethereum mainnet...
INFO [10-23|00:29:36.321] Maximum peer count                       ETH=30 LES=0 total=30
INFO [10-23|00:29:36.321] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [10-23|00:29:36.321] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable
Welcome to the Geth JavaScript console!

instance: Geth/v1.1.2-c4f93121-20210825/linux-amd64/go1.15.14
at block: 0 (Mon Apr 20 2020 21:46:54 GMT+0800 (+08))
 datadir: <my_data_dir_here>
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 parlia:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

To exit, press ctrl-d
> eth.syncing.
eth.syncing.constructor eth.syncing.toString eth.syncing.valueOf
> eth.syncing
{ 
  currentBlock: 0,
  highestBlock: 11997535,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 0
}
> net.peerCount
2
> net.peerCount
4
> panic: deletion not supported

goroutine 459309 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0, 0xc019bb5698, 0xc07dd149a0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:194 +0xd2
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000c780, 0xc061a85500)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:1928 +0x3f4
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000c780, 0xd206470f7c921f7b, 0x3ee4d8465b3ed0b8, 0xb21e6d5f47bdd0a1, 0xa01669445816a58b, 0xc01662c780, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:653 +0xc2e
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc02de3d7c0)
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:114 +0x1ef

Tried several times to do fresh sync.
Impossible to sync full node.
Any workaround?

Thanks

The text was updated successfully, but these errors were encountered:

vrogojin · 2021-10-23T10:51:37Z

After this crash, bash console freezes.
No I/O errors. No hardware errors.
What is going on?

vrogojin · 2021-10-23T10:59:37Z

More clarifications:
System information
Geth
Version: 1.1.2 (binary pre-build as of 22.10.2021)
Architecture: amd64
Operating System: Ubuntu 20.04.3 LTS

Expected behaviour
No panic.

Actual behaviour
It panics.

Steps to reproduce the behaviour
Run fresh geth for an hour, then it panics.
geth --syncmode=snap --config ./config.toml --datadir node --cache 18000 --rpc --http.port 9545 --http.corsdomain "*" --http.api "eth,web3,personal,net,txpool" --ws --ws.port 8546 --ws.origins "*" \ --ws.api "eth,web3,personal,net,txpool" --rpc.allow-unprotected-txs --txlookuplimit 0 console

Backtrace


panic: deletion not supported

goroutine 459309 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0, 0xc019bb5698, 0xc07dd149a0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:194 +0xd2
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000c780, 0xc061a85500)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:1928 +0x3f4
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000c780, 0xd206470f7c921f7b, 0x3ee4d8465b3ed0b8, 0xb21e6d5f47bdd0a1, 0xa01669445816a58b, 0xc01662c780, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:653 +0xc2e
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc02de3d7c0)
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:114 +0x1ef

grGred · 2021-10-26T13:01:33Z

System information:
Geth
Version: 1.1.3
Architecture: amd64
Go Version: go1.13.8
Operating System: linux

Expected behaviour:
No panic. Countinues to synchronize.

Actual behaviour:
It panics, getting a bunch of errors and stops. Happens only with syncmode = snap.

Steps to reproduce the behaviour:
Run geth with command:
./build/bin/geth --config ./config.toml --datadir ./node --cache 32000 --rpc.allow-unprotected-txs --txlookuplimit 0 --maxpeers 4000 --syncmode snap --diffsync --cache.database 70 --cache.trie 30
In about an hour panic occurs, geth stops with output:

panic: deletion not supported

goroutine 217727 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc16e2e6ce0, 0xc12101a460, 0x20, 0x20, 0x29f31d8, 0x0, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/trie/stacktrie.go:194 +0xd0
github.com/ethereum/go-ethereum/trie.VerifyRangeProof(0x5615df180157484d, 0x7279ef9f81c8591d, 0xd838d87ee3b6e45c, 0xe96fc751a40f659a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum@/trie/proof.go:486 +0x20f
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).OnStorage(0xc01bba8780, 0x1c723e0, 0xc09c34ed80, 0x622732924331ee0b, 0xc067a6c580, 0x1d, 0x1d, 0xc067a6c840, 0x1d, 0x1d, ...)  
        github.com/ethereum/go-ethereum@/eth/protocols/snap/sync.go:2423 +0x852
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).DeliverSnapPacket(0xc01bba8960, 0xc09c34ed80, 0x1c4bb60, 0xc10d944900, 0x2d6c7e01, 0xc10d944900)
        github.com/ethereum/go-ethereum@/eth/downloader/downloader.go:2044 +0x347
github.com/ethereum/go-ethereum/eth.(*snapHandler).Handle(0xc0001338c0, 0xc09c34ed80, 0x1c4bb60, 0xc10d944900, 0x3, 0x622732924331ee0b)
        github.com/ethereum/go-ethereum@/eth/handler_snap.go:49 +0x50
github.com/ethereum/go-ethereum/eth/protocols/snap.handleMessage(0x1c66720, 0xc0001338c0, 0xc09c34ed80, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:373 +0x21ea
github.com/ethereum/go-ethereum/eth/protocols/snap.handle(0x1c66720, 0xc0001338c0, 0xc09c34ed80, 0xc02403d700, 0xc0000a2b01)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:123 +0x43
github.com/ethereum/go-ethereum/eth/protocols/snap.MakeProtocols.func2.1(0xc09c34ed80, 0xc09c34ed80, 0x0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:103 +0x3d
github.com/ethereum/go-ethereum/eth.(*handler).runSnapExtension(0xc0001338c0, 0xc09c34ed80, 0xc182754980, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/eth/handler.go:374 +0x208
github.com/ethereum/go-ethereum/eth.(*snapHandler).RunPeer(0xc0001338c0, 0xc09c34ed80, 0xc182754980, 0xc06a61aa00, 0xc09c34ed80)
        github.com/ethereum/go-ethereum@/eth/handler_snap.go:33 +0x3f
github.com/ethereum/go-ethereum/eth/protocols/snap.MakeProtocols.func2(0xc0983f28a0, 0x1c4c2e0, 0xc06a61aa00, 0x0, 0xc04dcdc6c0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:102 +0xbb
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc0983f28a0, 0xc06a61aa00, 0x1c4c2e0, 0xc06a61aa00)

        github.com/ethereum/go-ethereum@/p2p/peer.go:396 +0xa3
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        github.com/ethereum/go-ethereum@/p2p/peer.go:394 +0x203

Logs doesn't show what was an actual problem:

t=2021-10-26T14:35:46+0300 lvl=info msg="State sync in progress"                 synced=41.48% state="56.41 GiB" accounts=29,722,[email protected] slots=239,049,[email protected] codes=184,[email protected] eta=21m54.622s

t=2021-10-26T14:35:48+0300 lvl=info msg="Imported new block headers"             count=1   elapsed=10.561ms    number=12,105,895 hash=0xe9ef0eae0aa3824d3739dda7c67d559abb8dcdc87e6308d5d057a472822d517c

t=2021-10-26T14:35:49+0300 lvl=eror msg="Snapshot extension registration failed" peer=98998f1a err="peer connected on snap without compatible eth support"

t=2021-10-26T14:35:50+0300 lvl=eror msg="Snapshot extension registration failed" peer=a23dedfc err="peer connected on snap without compatible eth support"

t=2021-10-26T14:35:51+0300 lvl=info msg="Imported new block headers"             count=2   elapsed=19.894ms    number=12,105,897 hash=0x3515ac3fe07c4ca6452a5fe4b37a04659181295fb8ff9a07a3f4a531d594fbed

t=2021-10-26T14:35:54+0300 lvl=info msg="State sync in progress"                 synced=41.50% state="56.51 GiB" accounts=29,737,[email protected] slots=239,510,[email protected] codes=184,[email protected] eta=21m49.253s

t=2021-10-26T14:35:55+0300 lvl=info msg="Imported new block headers"             count=1   elapsed=24.286ms    number=12,105,898 hash=0x68bfe126cf210995989b5738b6c0d4f9b44f19263fdf7efb4ebeef97c07ef31e

t=2021-10-26T14:35:55+0300 lvl=eror msg="Snapshot extension registration failed" peer=5e8011ec err="peer connected on snap without compatible eth support"

**UPD: **
Happend once again. A bit another output:

panic: deletion not supported

goroutine 387018 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc05f5add40, 0xc11e17a0a0, 0x20, 0x20, 0x29f31d8, 0x0, 0x0, 0xc0d4bcf728, 0xc0a2415e40)
        github.com/ethereum/go-ethereum@/trie/stacktrie.go:194 +0xd0
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc05f5add40, 0xc11e17a0a0, 0x20, 0x20, 0x29f31d8, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000cd20, 0xc0f6838c00)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/sync.go:1928 +0x3ea
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000cd20, 0xf5dab19f0d4d211d, 0x90df7080e18d0f79, 0x47dca2993be8cfdd, 0xacd7393842fe704f, 0xc0f35fc060, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/sync.go:653 +0xbfb
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc0707a2280)
        github.com/ethereum/go-ethereum@/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum@/eth/downloader/statesync.go:114 +0x1e9

webhype · 2021-10-26T17:36:36Z

Same here, but version 1.1.3 - any ideas?

grGred · 2021-10-26T17:48:47Z

Same here, but version 1.1.3 - any ideas?

My node crashed 3 times, and after that while starting i always recieved something like: Detected unclear exit.
After a some moment it stopped crashing. Now it compacts database and continues sync. So I guess you should just monitor it to resume in case of crashing.

princesscrypto1337 · 2021-10-30T17:42:18Z

I'm running into the same panic after syncing from scratch with snap mode

panic: deletion not supported

goroutine 1531641 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc053ffcf70, 0xc029140460, 0x20, 0x20, 0x280a478, 0x0, 0x0, 0xc043d8b7e8, 0xc0da2e3e00)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:194 +0xd2
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc053ffcf70, 0xc029140460, 0x20, 0x20, 0x280a478, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000cf00, 0xc04c478600)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:1928 +0x3f4
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000cf00, 0xcf8fef4722d1f77f, 0x1753f77344dc40fd, 0xaf97cc4260622dc9, 0xd2d6d8a0580567aa, 0xc019de17a0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:653 +0xcae
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc0582ffae0)
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:114 +0x1ef

b10c77 · 2021-10-31T10:22:00Z

Seems to be related to syncing using snap mode, don't get this with the default method but got when using snap

Marijus · 2021-11-09T12:10:24Z

facing same issue - any tips?

vrogojin · 2021-11-09T21:14:35Z

I switched to other physical nvme, and it worked without problem. The first nvme drive itself was fine though. No idea, what geth did not like about it.

MischaWeerwag · 2021-11-15T14:49:47Z

Same issue. Looks like it can be fixed by increasing the soft/hard limit of open files from 1024 to 65535.

Edit, looks like it isnt completely gone but significantly reduced.

ikgo · 2021-12-04T15:22:01Z

any status update on the bug.
my node v1.1.5 enter to restart loop. panic after couple of minutes after star syncing.

prune-state exit with message

ERROR[12-04|15:03:04.063] Failed to prune state                    err="snapshot not old enough yet: need 128 more blocks"
snapshot not old enough yet: need 128 more blocks

is a version 1.1.6 fix this bug?

seaify · 2021-12-07T12:14:09Z

when can it be fixed? can't sync

keefel · 2021-12-16T04:02:05Z

Fixed by #667, please wait for next release.

vrogojin mentioned this issue Nov 3, 2021

crashed with "panic: deletion not supported" #505

Closed

keefel added the bug Something isn't working label Dec 3, 2021

keefel mentioned this issue Dec 16, 2021

trie: reject deletions when verifying range proofs #667

Merged

keefel closed this as completed Dec 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic: deletion not supported #475

panic: deletion not supported #475

vrogojin commented Oct 23, 2021

vrogojin commented Oct 23, 2021

vrogojin commented Oct 23, 2021

grGred commented Oct 26, 2021 •

edited

Loading

webhype commented Oct 26, 2021

grGred commented Oct 26, 2021 •

edited

Loading

princesscrypto1337 commented Oct 30, 2021

b10c77 commented Oct 31, 2021

Marijus commented Nov 9, 2021

vrogojin commented Nov 9, 2021

MischaWeerwag commented Nov 15, 2021 •

edited

Loading

ikgo commented Dec 4, 2021 •

edited

Loading

seaify commented Dec 7, 2021

keefel commented Dec 16, 2021

panic: deletion not supported #475

panic: deletion not supported #475

Comments

vrogojin commented Oct 23, 2021

vrogojin commented Oct 23, 2021

vrogojin commented Oct 23, 2021

grGred commented Oct 26, 2021 • edited Loading

webhype commented Oct 26, 2021

grGred commented Oct 26, 2021 • edited Loading

princesscrypto1337 commented Oct 30, 2021

b10c77 commented Oct 31, 2021

Marijus commented Nov 9, 2021

vrogojin commented Nov 9, 2021

MischaWeerwag commented Nov 15, 2021 • edited Loading

ikgo commented Dec 4, 2021 • edited Loading

seaify commented Dec 7, 2021

keefel commented Dec 16, 2021

grGred commented Oct 26, 2021 •

edited

Loading

grGred commented Oct 26, 2021 •

edited

Loading

MischaWeerwag commented Nov 15, 2021 •

edited

Loading

ikgo commented Dec 4, 2021 •

edited

Loading