statedb: allow longer state pruning history #11980

NingLin-P · 2022-08-04T23:01:56Z

Fixes #11911

This PR introduces a database-backed queue for the state pruning window, which only keeps a few blocks in memory and loads more blocks on demand, thus reducing the burden of setting a long state pruning history.

Note that the queue required the backend database to support reference counting (i.e parity-db), databases that do not support reference counting (i.e rocksdb) still need to keep all blocks in memory

polkadot address: 1CWyso9Zw46QdR54sTvKJXTUdmZznYZa2aJfNHdG6xQ6L71

Signed-off-by: linning <[email protected]>

cla-bot-2021 · 2022-08-04T23:02:02Z

User @NingLin-P, please sign the CLA here.

NingLin-P · 2022-08-04T23:04:22Z

PTAL @arkpar

Signed-off-by: linning <[email protected]>

arkpar · 2022-08-05T06:33:08Z

Thank you for the PR! Consider adding your polkadot address in the format described here.

For now I only took a basic look and it looks mostly good. There's just one issue. I don't think there's any real need to maintain DbBacked::hashs. The problem with hashs is that whie the memory requirement is much smaller, it is still O(n). The bigger problem is that it requires loading all journal records on startup, which can take a long time with millions of records.

We can always get the next hash to prune from the database by loading the next journal record. For the next_hash to work, we can preload the next record. So instead of hashs we can maintain next which is a preloaded JournalRecord for the next block to pune. When it is time to prune a block 'N', we just use the data from next and then preload the next block ('N' + 1) from the database into next.

NingLin-P · 2022-08-05T10:06:29Z

There's just one issue. I don't think there's any real need to maintain DbBacked::hashs. The problem with hashs is that whie the memory requirement is much smaller, it is still O(n). The bigger problem is that it requires loading all journal records on startup, which can take a long time with millions of records.

Indeed, DbBacked::hashs still have these problems but the real need to maintain DbBacked::hashs is RefWindow::have_block (we should replace hashs: VecDeque with BTreeSet though).

I have also checked the upper callers that required RefWindow::have_block but it seems hard to remove DbBacked::hashs without other ways to determine if a block is canonicalized (then we can implement have_block just by checking the block number)

arkpar · 2022-08-05T11:06:43Z

I have also checked the upper callers that required RefWindow::have_block but it seems hard to remove DbBacked::hashs without other ways to determine if a block is canonicalized (then we can implement have_block just by checking the block number)

All the block in RefWindow are guaranteed to be canonicalized. Blocks that are not yet canonicalized are handled by NonCanonicalOverlay. So that's not an issue.

have_block is surrently used in

is_pruned - we can check by number here.
pin - it is fine to just remove the check and always add to pinned set.

NingLin-P · 2022-08-05T11:37:14Z

All the block in RefWindow are guaranteed to be canonicalized

But the questioning block may not be canonicalized (like the parameter pass to is_pruned), the calling chain of is_pruned involved a lot of functions of the sc_client_api::backend::Backend trait implement for sc_client_db::Backend (i.e like Backend::state_at)

arkpar · 2022-08-05T12:08:46Z

is_pruned and pin already check if the block is in the non-canonicalized overlay by calling NonCanonicalOverlay::have_block. Not to be confused with RefWindow::have_block. That's a different have_block function and there's no need to change it.

The block passed to is_pruned, or rather have_state_at may be in one of the following states:

Pending canonicalization - it should be in NonCanonicalOverlay and is_pruned will return it here:

substrate/client/state-db/src/lib.rs

Line 355 in 9251791

!self.non_canonical.have_block(hash)
Pending pruning. This may be checked by number as long as the header is known.
Already pruned. This is the case where the header is in the db, but the number is out of the pruning window.
Unknow, as in was never inserted. This is when the header is missing.

So I suppose we'd need to document that is_pruned and pin should ony be called for blocks with existing headers and add an additional check that the header exists in have_state_at and it should be fine.

For additional safety I'd add a number argument to pin and check that the number is in the ref window there as well.

Signed-off-by: linning <[email protected]>

NingLin-P · 2022-08-05T20:16:39Z

@arkpar Thanks for the explanation! I have changed the implementation, but two tests of is_pruned is broken, will fix it later.

Signed-off-by: linning <[email protected]>

NingLin-P · 2022-08-08T15:15:43Z

PTAL again @arkpar

arkpar · 2022-08-08T17:51:58Z

After some consideration I can see that is_pruned/pin is still a bit problematic, considering that the caller ineed does not know when the hash is taken into account or the number. I think we can fix this with the following:

pin should return an enum

Pinned when the ginven hash and number are definietely known to the state DB. This is when the block is pending canonicalization, or when the hash is in the in-mem backend for RefWindow or in the cache.
MaybePinned this is when the block with given number is in the pruning window, but can't verify that the hash matches. MaybePinned should also be returned when archive mode is enabled. When this is returned, the backend in sc-client-db should do an additonal check for state root, like this:

substrate/client/db/src/lib.rs

Lines 2286 to 2290 in d042666

Ok(header) => sp_state_machine::Storage::get(

self.storage.as_ref(),

&header.state_root,

(&[], None),

)
Err(..) When we know that the block definietely does not exist.

is_pruned should have a similar enum return value with similar check.

A call to have_state_at before pin should be removed. This use case is racy anyway. You can't do anything useful with the value returned by have_state_at since some other thread my cause pruning to happen and the result is obsolete right after the call. The only use for have_state_at should be for the cases when the result being up to date is not that important. E.g. when showing block status to the user.

Signed-off-by: linning <[email protected]>

NingLin-P · 2022-08-09T10:18:03Z

Comments addressed

pin is implemented a bit differently, because pin need to pin the block according to the have_block result immediately, so we can't return a MaybePin to the caller to do further checking then pin again. I pass a closure to pin instead, which will do the further checking inside pin if needed.

NingLin-P · 2022-08-09T16:40:13Z

Perhaps we can handle the MayHaveBlock case in a simpler way. Currently, with the block number passing to RefWindow::have_block we can load the exact single journal record from db and check if its hash matches the hash parameter (thanks to the block number we don't need to iterate over the whole queue to find the hash as the previous implementation does).

Since the sp_state_machine::Storage::get may also result in a db read, this approach does not bring much cost but is much simpler.

arkpar · 2022-08-09T17:54:01Z

pin is implemented a bit differently, because pin need to pin the block according to the have_block result immediately, so we can't return a MaybePin to the caller to do further checking then pin again. I pass a closure to pin instead, which will do the further checking inside pin if needed.

I was imagning that MaybePinned actuall places the pin, and if the additional check fails, unpin should be called. But closure works as well. pin should be made to return a guard really, but that's out of scope of this PR.

Perhaps we can handle the MayHaveBlock case in a simpler way. Currently, with the block number passing to RefWindow::have_block we can load the exact single journal record from db and check if its hash matches the hash parameter (thanks to the block number we don't need to iterate over the whole queue to find the hash as the previous implementation does).

The difference is how much data needs to be queried from the DB. Single root trie node is small, but the journal record may contain a lot of keys. An alternative storage scheme would me more efficient, but I don't think it is worth the effort to change it now.

client/state-db/src/lib.rs

client/state-db/src/pruning.rs

cheme · 2022-08-10T19:02:38Z

Actually, the pruning window is a FIFO queue now, blocks are put to the back (import ) and pop out from the head (prune_one and apply_pending). The cache is only a slice of successive blocks at the head of the window (it is also a FIFO queue), to reduce the burden of keeping all block in memory thus allow longer state pruning window.

yes fifo was wrong wording from me, what I meant was in your code you give priority to have first out block over first in; opposite could be done. The fact that we force consecutive range makes it we are emptying back of vecdeque, batch refill it, emptying, and so on, so one idea would be to just keep index of block and not force consecutive range (avoid batch of query this way). But that is only for when the pruning range don't fit in memory.

NingLin-P · 2022-08-10T19:52:10Z

so one idea would be to just keep index of block and not force consecutive range (avoid batch of query this way). But that is only for when the pruning range don't fit in memory.

Oh, the code emphasizes consecutive a lot is because blocks are pruning in consecutive order. The pruning window is a FIFO queue, new canonicalized block put in the back of the queue, when the queue's size is exceed the configured size, the block in the head is pop out and pruned until the size <= the configured size again. Since the blocks in the pruing window is only used for pruning, we only load the block into cache when it is gonna be pruned.

cheme · 2022-08-11T08:02:59Z

I think I looked in the wrong direction, what matters is to have one for the case where the window is in the default size (I believe it is 256 from

substrate/client/state-db/src/lib.rs

Line 269 in a1a9b47

Self { max_blocks: Some(256), max_mem: None }

, should be exposed and reuse as a constant).
I was looking at what happen when we got bigger window than the cache, and then just not having the active caching is also fine.

cheme · 2022-08-11T08:17:47Z

client/state-db/src/pruning.rs

 const PRUNING_JOURNAL: &[u8] = b"pruning_journal";
+// Default pruning window size plus a magic number keep most common ops in cache
+const CACHE_BATCH_SIZE: usize = 256 + 10;


We could expose 256 as another constant.
Also this size could be calculated by actual constraint value in state-db/src/lib.rs and not be constant.
Also, if the state-db do have a constraint > 256 (--pruning-size param), we could then just disable the cache and always fetch from db on pop_front.
This is a alternative, just save some cache memory and avoid the batch download.

We could expose 256 as another constant.
Also this size could be calculated by actual constraint value in state-db/src/lib.rs and not be constant.

Yes, we can calculate this cache size from the default window size, but overall we still need a constant to determine the max cache size.

Also, if the state-db do have a constraint > 256 (--pruning-size param), we could just disable the cache and always fetch from db on pop_front.
This is a alternative, just save some cache memory and avoid the batch download.

Having a cache will be helpful. Since all blocks are eventually needed to prune (thus need to load from db if window size larger than the max cache size), batch loading doesn't bring more cost.

And cache may be more friendly to the underlying caching (i.e page cache) as cache items are small (block hash + delete keys which is also hash I think) multiple items can fit in the same page cache. Also having a cache can avoid re-loading the block in case of revert_pending and it also makes the code clean as we don't need to add
the constraint > 256 condition around.

Having a cache will be helpful. Since all blocks are eventually needed to prune (thus need to load from db if window size larger than the max cache size), batch loading doesn't bring more cost.

The point of not having it is mainly to reduce code (maybe the query of 256 item would not be visible indeed).
As mention before, my opinion (may not be correct) on cache for pruning windows > cache size would be to use a sparse array (so not all item will have to be loaded/read from db). But really not a concern since first goal of this cache seems to be to maintain similar perf when using reasonable pruning window.

Signed-off-by: linning <[email protected]>

cheme

I think it is my last batch of comments 👍 ( I renounce changing your mind on the queue design)

client/db/src/lib.rs

client/state-db/src/pruning.rs

cheme · 2022-08-12T15:10:27Z

client/state-db/src/pruning.rs

+ // NOTE: `pop_front` should not return `MetaDb::Error` because blocks are visited
+ // by `RefWindow::prune_one` first then `RefWindow::apply_pending` and
+ // `DeathRowQueue::get` should load the blocks into cache already
+ .expect("block must loaded in cache thus no MetaDb::Error")


The NOTE does not make much sense to me, for me we just do not expect to have a missing journal and queue (queue and uncached) to be empty if pending_pruning is 0. (and pending pruning to be out of queue but this is next expectaiton)

pop_front return Result<Option<DeathRow<BlockHash, Key>>, Error<D::Error>>, the first expect is for the potential loading from db thus brings potential Error<D::Error> which is unexpected because there should no db loading for pop_front in the current usage as the comment explained. And the second expect is for the case you mentioned.

client/state-db/src/lib.rs

…runing

Signed-off-by: linning <[email protected]>

NingLin-P · 2022-08-14T18:21:19Z

client/state-db/src/pruning.rs

+ if self.base == 0 && self.queue.len() == 0 && number > 0 {
+ // assume that parent was canonicalized
+ self.base = number;
+ } else if (self.base + self.queue.len() as u64) != number {
+ return Err(Error::StateDb(StateDbError::InvalidBlockNumber))
+ }


This change is subject to:

substrate/client/state-db/src/noncanonical.rs

Lines 251 to 258 in 055453e

if self.levels.is_empty() && self.last_canonicalized.is_none() && number > 0 {

// assume that parent was canonicalized

let last_canonicalized = (parent_hash.clone(), number - 1);

commit

.meta

.inserted

.push((to_meta_key(LAST_CANONICAL, &()), last_canonicalized.encode()));

self.last_canonicalized = Some(last_canonicalized);

namely that the first block put into the pruning window may not be block 0. So in the previous implementation the block number in the pruning window is incorrect, but it is okay because its public functions didn't take any block number as parameter (perhaps this is why the previous have_block needs to brute force search all blocks)

But this PR optimized have_block by using block number indexing the query block thus causing the syncs_state test to fail, thus we need add this change to keep consistent with the non_canonical layer. Also, I have reverted the change in sc_state_db::tests::make_test_db (previously mentioned in #11980 (comment)) to add test cases (along with syncs_state) to this change.
/cc @arkpar @cheme

@NingLin-P , not sure if related, but I tried to run polkadot patched with this PR and prune seems to infiniloop ("Trying to prune when there's nothing to prune").

Do you have any log (trace level), I'd like to have a check

https://gist.github.com/cheme/f1ea299959f23d94636160cb00ace674
But actually does not seems related to your pr, there is something else I need to find (this only happen with parity-db).
Edit: well master with parity-db works fine so it should relate to this pr changes in some way

Ok, I am stupid!!! rocksdb do not use the DB backend. So yes trace attached possibly shows an issue with this PR.
Got them with ./target/release/polkadot -d ./tmp1 --state-pruning 256 --log state-db=trace --log client-db=trace 2&> log
Note that starting with a window of 900 blocks, I don't see the issue anymore (maybe using force_canonicalize ?).
Edit: actually seems more likely we are trying to get a key from db that is still in commit_set. (I see insert at some point and miss 1 millis after).

actually seems more likely we are trying to get a key from db that is still in commit_set. (I see insert at some point and miss 1 millis after).

Yes, I also found the cause of this issue is db return None for some blocks that had previously been added to the commit_set.

I was thinking fix this issue by

Adding an additional cache of recent added blocks, so we can get blocks that not yet commit to the database from this cache

Or, don't count block that not yet commit to the database for window_size, so we will not prune block that not yet commit to the database

but both approach required to do some cleanup work (clear cache or adjust window_size) in apply_pending/revert_pending, and seems these functions may be called concurrently which may mix up the state:

substrate/client/db/src/lib.rs

Lines 1940 to 1949 in bdd1399

match self.try_commit_operation(operation) {

Ok(_) => {

self.storage.state_db.apply_pending();

Ok(())

},

e @ Err(_) => {

self.storage.state_db.revert_pending();

e

},

}

For example, two CommitSet c1 and c2, apply_pending is called for c2 first, if later revert_pending need to call for c1 there is nothing left to revert since previous apply_pending alreay apply all the change (not sure if this problem exist in master branch).

Adding some counter state to identify which CommitSet the apply_pending/revert_pending is calling for may workaround this problem, but it is still tricky to recover from case like some new blocks success commit to db but some older blocks fail.

That's an issue I did not foresee :(
A third possibility would be to just not prune and prune more when available in db.
On prune_one, we prune from last pruned to last canonicalized - range and skip if not available (maybe it requires to maintain a new last_pruned counter in db, and maybe something else for warp synch).
@arkpar, this issue makes me wonder if there may be some unpruned content, I mean if currently we may prune over content that is still in a commit set during synch.

For example, two CommitSet c1 and c2, apply_pending is called for c2 first, if later revert_pending need to call for c1 there is nothing left to revert since previous apply_pending alreay apply all the change (not sure if this problem exist in master branch).

commit_operatoin only happens in one thread. There's a higher level lock in the Client that synchronises block import operations. This low level API is prone to misuse indeed. I'm considering removing the pending state alltogether and simply re-create the statedb object to recover from error.

@arkpar, this issue makes me wonder if there may be some unpruned content, I mean if currently we may prune over content that is still in a commit set during synch.

This should not be happening. Blocks are imported one at a time and we only prune finalized blocks.

The issue is fixed, PTAL again.

…runing

Signed-off-by: linning <[email protected]>

cheme

LGTM, a few question, but nothing blocking.

I ran a few synch tests and got an error if starting a new chain with "--pruning 0" but I think we can ignore it.

cheme · 2022-08-26T09:34:01Z

client/db/src/lib.rs

+ )
+ .unwrap_or(None)
+ .is_some(),
+ _ => false,


Here I think it would be better to return a ClientError (have_state_at seems to be only call by function how return it).

Changing the return type here will also require changing the Backend interface, some callers of it may not expect an error

I vote for changing it (client db is quite internal to substrate I think). But let's wait for second feed back.

client/state-db/src/lib.rs

client/state-db/src/pruning.rs

cheme · 2022-08-26T13:05:12Z

client/state-db/src/pruning.rs

+
+/// The result return by `RefWindow::have_block`
+#[derive(Debug, PartialEq, Eq)]
+pub enum HaveBlock {


Option instead of HaveBlock or IsPruned would be less expressive, but would make code a bit shorter (using map and unwrap_or).
Not sure if better, just mentioning that I am not against it.

Indeed, it is a trade off hard to make

client/state-db/src/pruning.rs

cheme · 2022-08-26T14:16:16Z

client/state-db/src/pruning.rs

- let new_max_block = self.death_rows.len() as u64 + self.pending_number;
- self.death_index.retain(|_, block| *block < new_max_block);
- }
+ self.queue.revert_recent_add(self.base, self.pending_canonicalizations);
 self.pending_canonicalizations = 0;
 self.pending_prunings = 0;


@arkpar not exactly sure why we change pending_pruning here (was in the original code).

I thought the reason may be the CommitSet that has the data collected in prune_one is fail to commit to the db, reset the pending_prunings could allow us to re-collect the data in prune_one and retry.

Signed-off-by: linning <[email protected]> Co-authored-by: cheme <[email protected]>

Signed-off-by: linning <[email protected]>

arkpar · 2022-09-01T13:47:09Z

bot merge

paritytech-processbot · 2022-09-01T13:47:20Z

Waiting for commit status.

rvalle · 2022-12-02T08:33:43Z

@NingLin-P and @arkpar I am testing this. I can see the effect in memory but I can not query more than 4096 blocks still, I have filled paritytech/polkadot#6378 while trying to understand what is going on.

* introduce DbBackedQueue for the state pruning window Signed-off-by: linning <[email protected]> * avoid cloning for next_hash Signed-off-by: linning <[email protected]> * add tests Signed-off-by: linning <[email protected]> * make clippy happy Signed-off-by: linning <[email protected]> * impl have_block by checking block number Signed-off-by: linning <[email protected]> * refactor Signed-off-by: linning <[email protected]> * fix tests & add test for init db-backed queue Signed-off-by: linning <[email protected]> * update comment Signed-off-by: linning <[email protected]> * add check for have_state_at Signed-off-by: linning <[email protected]> * address comment Signed-off-by: linning <[email protected]> * renanme unload_blocks to uncached_blocks Signed-off-by: linning <[email protected]> * address comment Signed-off-by: linning <[email protected]> * fix syncs_state test Signed-off-by: linning <[email protected]> * address comment Signed-off-by: linning <[email protected]> * revert change to make_test_db to add test cases Signed-off-by: linning <[email protected]> * do not prune unavailable block & add tests Signed-off-by: linning <[email protected]> * Update client/state-db/src/lib.rs Signed-off-by: linning <[email protected]> Co-authored-by: cheme <[email protected]> * Update client/state-db/src/pruning.rs Signed-off-by: linning <[email protected]> Co-authored-by: cheme <[email protected]> * address comment Signed-off-by: linning <[email protected]> Signed-off-by: linning <[email protected]> Co-authored-by: cheme <[email protected]>

NingLin-P added 3 commits August 5, 2022 02:29

introduce DbBackedQueue for the state pruning window

ff20ef7

Signed-off-by: linning <[email protected]>

avoid cloning for next_hash

a0280fa

Signed-off-by: linning <[email protected]>

add tests

b05f699

Signed-off-by: linning <[email protected]>

make clippy happy

e8d57d0

Signed-off-by: linning <[email protected]>

NingLin-P added 2 commits August 6, 2022 00:14

impl have_block by checking block number

f048c31

Signed-off-by: linning <[email protected]>

refactor

206582e

Signed-off-by: linning <[email protected]>

NingLin-P added 3 commits August 8, 2022 19:25

fix tests & add test for init db-backed queue

bc3e31c

Signed-off-by: linning <[email protected]>

update comment

1944c5c

Signed-off-by: linning <[email protected]>

add check for have_state_at

d042666

Signed-off-by: linning <[email protected]>

address comment

869eb5d

Signed-off-by: linning <[email protected]>

arkpar reviewed Aug 9, 2022

View reviewed changes

client/state-db/src/lib.rs Outdated Show resolved Hide resolved

arkpar reviewed Aug 9, 2022

View reviewed changes

client/state-db/src/lib.rs Outdated Show resolved Hide resolved

arkpar reviewed Aug 9, 2022

View reviewed changes

client/state-db/src/lib.rs Show resolved Hide resolved

arkpar reviewed Aug 9, 2022

View reviewed changes

client/state-db/src/lib.rs Show resolved Hide resolved

arkpar reviewed Aug 9, 2022

View reviewed changes

client/state-db/src/lib.rs Outdated Show resolved Hide resolved

arkpar reviewed Aug 9, 2022

View reviewed changes

client/state-db/src/pruning.rs Outdated Show resolved Hide resolved

arkpar requested a review from bkchr August 9, 2022 18:35

cheme reviewed Aug 11, 2022

View reviewed changes

address comment

abf1354

Signed-off-by: linning <[email protected]>

cheme reviewed Aug 12, 2022

View reviewed changes

NingLin-P added 4 commits August 13, 2022 01:53

Merge branch 'master' of github.com:paritytech/substrate into state-p…

0c0a8eb

…runing

fix syncs_state test

16969d1

Signed-off-by: linning <[email protected]>

address comment

7c886f7

Signed-off-by: linning <[email protected]>

revert change to make_test_db to add test cases

acfa7ef

Signed-off-by: linning <[email protected]>

NingLin-P commented Aug 14, 2022

View reviewed changes

NingLin-P added 2 commits August 15, 2022 04:33

Merge branch 'master' of github.com:paritytech/substrate into state-p…

ba46914

…runing

do not prune unavailable block & add tests

77c830a

Signed-off-by: linning <[email protected]>

cheme approved these changes Aug 26, 2022

View reviewed changes

NingLin-P and others added 3 commits August 29, 2022 17:32

Update client/state-db/src/lib.rs

30a1f1e

Signed-off-by: linning <[email protected]> Co-authored-by: cheme <[email protected]>

Update client/state-db/src/pruning.rs

917e26e

Signed-off-by: linning <[email protected]> Co-authored-by: cheme <[email protected]>

address comment

469b0b2

Signed-off-by: linning <[email protected]>

arkpar approved these changes Sep 1, 2022

View reviewed changes

paritytech-processbot bot merged commit 6bd6b31 into paritytech:master Sep 1, 2022

arkpar mentioned this pull request Sep 12, 2022

State-db refactoring #12239

Merged

rvalle mentioned this pull request Dec 2, 2022

Large pruning parameter still not honored despite substrate fix. paritytech/polkadot#6378

Closed

jasl mentioned this pull request Dec 13, 2022

Khala Node get stuck at a block height and Pherry fails to connect to node Phala-Network/phala-blockchain#1070

Closed

	if self.levels.is_empty() && self.last_canonicalized.is_none() && number > 0 {
	// assume that parent was canonicalized
	let last_canonicalized = (parent_hash.clone(), number - 1);
	commit
	.meta
	.inserted
	.push((to_meta_key(LAST_CANONICAL, &()), last_canonicalized.encode()));
	self.last_canonicalized = Some(last_canonicalized);

	match self.try_commit_operation(operation) {
	Ok(_) => {
	self.storage.state_db.apply_pending();
	Ok(())
	},
	e @ Err(_) => {
	self.storage.state_db.revert_pending();
	e
	},
	}

statedb: allow longer state pruning history #11980

statedb: allow longer state pruning history #11980

Conversation

NingLin-P commented Aug 4, 2022 • edited Loading

cla-bot-2021 bot commented Aug 4, 2022

NingLin-P commented Aug 4, 2022

arkpar commented Aug 5, 2022

NingLin-P commented Aug 5, 2022

arkpar commented Aug 5, 2022

NingLin-P commented Aug 5, 2022

arkpar commented Aug 5, 2022

NingLin-P commented Aug 5, 2022

NingLin-P commented Aug 8, 2022

arkpar commented Aug 8, 2022

NingLin-P commented Aug 9, 2022 • edited Loading

NingLin-P commented Aug 9, 2022

arkpar commented Aug 9, 2022

cheme commented Aug 10, 2022

NingLin-P commented Aug 10, 2022

cheme commented Aug 11, 2022

cheme Aug 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheme Aug 18, 2022 • edited Loading

Choose a reason for hiding this comment

cheme Aug 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P Aug 29, 2022 • edited Loading

Choose a reason for hiding this comment

arkpar commented Sep 1, 2022

paritytech-processbot bot commented Sep 1, 2022

rvalle commented Dec 2, 2022 • edited Loading

NingLin-P commented Aug 4, 2022 •

edited

Loading

NingLin-P commented Aug 9, 2022 •

edited

Loading

cheme Aug 11, 2022 •

edited

Loading

cheme Aug 18, 2022 •

edited

Loading

cheme Aug 18, 2022 •

edited

Loading

NingLin-P Aug 29, 2022 •

edited

Loading

rvalle commented Dec 2, 2022 •

edited

Loading