Warnings about discarded state.

[eskimor]

On Westend and reproducible on Versi, we see warnings about state already being discarded:

  | 2022-06-27 15:25:26 | 2022-06-27 13:25:26.250  WARN tokio-runtime-worker parachain::runtime-api: cannot query the runtime API version: UnknownBlock: State already discarded for BlockId::Hash(0xcc5b95fe2c408170bb864c5f13f84d77fa9fe4c562650c9985f4850425d9f5a7)
-- | -- | --

If we look at other logs in that time range, we can see that we are getting quite a few out-of-sync messages as well:

Explore-logs-2022-06-27 18 16 15.txt

So the reason could be that nodes are just out of sync ... due to heavy load? Other reasons?

[bkchr]

Validators are not running as archive nodes anymore. Default pruning window is 250 blocks after the last finalized block. Maybe you are accessing old data?

[eskimor]

Those warnings together with these:

• parachain::availability-distribution: fetch_pov_job err=FetchPoV(NetworkError(NotConnected))
• tokio-runtime-worker wasm-runtime: Ran out of free WASM instances

are present since at least 0.9.23.

[arkpar]

Is 0xcc5b95fe2c408170bb864c5f13f84d77fa9fe4c562650c9985f4850425d9f5a7 in the finalized chain?

[eskimor]

When querying the rpc node for the hash, I also get that discarded message: https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fversi-rpc-node-0.parity-versi.parity.io#/explorer/query/0xcc5b95fe2c408170bb864c5f13f84d77fa9fe4c562650c9985f4850425d9f5a7

I guess that does not mean much, as that block is very old now. What is a good way to check that? I don't think we have subscan and such for Versi. I could write some code to query the chain API ... other options?

[arkpar]

It matters because there are two possible reasoins for this message.

1. Block is too old and discarded because of pruning, as @bkchr sugggests
2. Block was discarded because it part of a fork that was not finalized in the first place.

An easy way to check that would be to make an RPC call to get the header by that hash and note the block number. Then make an RPC call to get a block hash by that number and check if it matches the hash in question.
Unfortunatell when I try to call a getHeader RPC here, I get the same error messge. Which is really weird because getHeader should not really create a state object and access the state. Any idea why it does @bkchr ?

In any case, it seems like this particular hash is too old indeed. So I'd add more logs to figure out why candidate backing tries to validate 250 blocks behind the tip of the chain.

[bkchr]

Any idea why it does @bkchr ?

After getting the db and playing around I realized what the problem is. Polkadot-js is calling runtime_version for each rpc call. If you call the rpc manually, it works.

[eskimor]

Potential duplicate of paritytech/polkadot-sdk#793