[StateService] Verification #470

ZhangTao1596 · 2021-01-18T10:48:48Z

Step 2/3 StateService

Since there are too many changes in StateService, I will split it into 3 parts to commit pull request, and this is the 2nd.

Storage and P2P message (Merged)
Verification
More command & RPC

Changes:

Add Verification service to get consensus of state root from verifiers.
- Cache `MaxCachedValidationProcess· Verification processes because block grows without state root.
- Node use latest state root to compare state.
- Not consensused state root of height will be dropped when block grows.
Add RPC method to exchange consensus message.
Add command to start Verification service.

src/StateService/Settings.cs

src/StateService/StatePlugin.cs

src/StateService/Validation/ValidationProcess.cs

src/StateService/StatePlugin.cs

src/StateService/Verification/VerificationContext.cs

shargon · 2021-01-31T09:46:42Z

src/StateService/Verification/VerificationContext.cs

+using Akka.Actor;
+using Neo.Cryptography;
+using Neo.Cryptography.ECC;
+using Neo.IO;
+using Neo.Ledger;
+using Neo.Network.P2P;
+using Neo.Network.P2P.Payloads;
+using Neo.Persistence;
+using Neo.Plugins.StateService.Storage;
+using Neo.SmartContract;
+using Neo.SmartContract.Native;
+using Neo.Wallets;
+using System;
+using System.Collections.Generic;
+
+namespace Neo.Plugins.StateService.Verification
+{
+ public class VerificationContext
+ {
+ private const uint MaxValidUntilBlockIncrement = 100;
+ private StateRoot root;
+ private ExtensiblePayload payload;
+ private readonly Wallet wallet;
+ private readonly KeyPair keyPair;
+ private readonly int myIndex;
+ private readonly uint rootIndex;
+ private readonly ECPoint[] verifiers;
+ private int M => verifiers.Length - (verifiers.Length - 1) / 3;
+ private readonly Dictionary<int, byte[]> signatures = new Dictionary<int, byte[]>();


Suggested change

using Akka.Actor;

using Neo.Cryptography;

using Neo.Cryptography.ECC;

using Neo.IO;

using Neo.Ledger;

using Neo.Network.P2P;

using Neo.Network.P2P.Payloads;

using Neo.Persistence;

using Neo.Plugins.StateService.Storage;

using Neo.SmartContract;

using Neo.SmartContract.Native;

using Neo.Wallets;

using System;

using System.Collections.Generic;

namespace Neo.Plugins.StateService.Verification

{

public class VerificationContext

{

private const uint MaxValidUntilBlockIncrement = 100;

private StateRoot root;

private ExtensiblePayload payload;

private readonly Wallet wallet;

private readonly KeyPair keyPair;

private readonly int myIndex;

private readonly uint rootIndex;

private readonly ECPoint[] verifiers;

private int M => verifiers.Length - (verifiers.Length - 1) / 3;

private readonly Dictionary<int, byte[]> signatures = new Dictionary<int, byte[]>();

using Akka.Actor;

using Neo.Cryptography;

using Neo.Cryptography.ECC;

using Neo.IO;

using Neo.Ledger;

using Neo.Network.P2P;

using Neo.Network.P2P.Payloads;

using Neo.Persistence;

using Neo.Plugins.StateService.Storage;

using Neo.SmartContract;

using Neo.SmartContract.Native;

using Neo.Wallets;

using System;

using System.Collections.Concurrent;

namespace Neo.Plugins.StateService.Verification

{

public class VerificationContext

{

private const uint MaxValidUntilBlockIncrement = 100;

private StateRoot root;

private ExtensiblePayload payload;

private readonly Wallet wallet;

private readonly KeyPair keyPair;

private readonly int myIndex;

private readonly uint rootIndex;

private readonly ECPoint[] verifiers;

private int M => verifiers.Length - (verifiers.Length - 1) / 3;

private readonly ConcurrentDictionary<int, byte[]> signatures = new ConcurrentDictionary<int, byte[]>();

Use concurrent?

Why use concurrent, it is only in one actor.

VerificationService use Concurrent for his dictionary and it use this class

I don't think we need concurrent in VerificationService, how do you think?

shargon · 2021-01-31T09:47:41Z

src/StateService/Verification/VerificationContext.cs

+ signatures.Add(index, sig);
+ return true;


Suggested change

signatures.Add(index, sig);

return true;

return signatures.TryAdd(index, sig);

src/StateService/Verification/VerificationContext.cs

src/StateService/Verification/VerificationService.cs

superboyiii · 2021-01-31T13:54:28Z

@erikzhang @shargon Please review again.

ZhangTao1596 · 2021-01-31T14:17:10Z

Merge?

vncoelho

There is something strange,
Once I activate this plugin the screen do not allow to be stopped on linux server with ctrl+c.
Multiple nodes are running together and the ones with this plugin activated suffers this.

src/StateService/Verification/VerificationService.cs

src/StateService/StateService/config.json

ZhangTao1596 · 2021-02-01T02:17:38Z

There is something strange,
Once I activate this plugin the screen do not allow to be stopped on linux server with ctrl+c.
Multiple nodes are running together and the ones with this plugin activated suffers this.

Did you install oracle plugin? As @ProDog tested #494, orcale plugin will cause this can't exit problem.

shargon · 2021-02-01T08:30:57Z

src/StateService/Verification/VerificationService.cs

+ {
+ if (MaxCachedVerificationProcessCount <= contexts.Count)
+ {
+ var indexes = contexts.Keys.OrderBy(i => i).ToArray();


I think ReadOnlySpan<T> is better.

I think ReadOnlySpan<T> is better.

Use this var indexes = new ReadOnlySpan<uint>(contexts.Keys.OrderBy(i => i).ToArray());?

if (MaxCachedVerificationProcessCount <= contexts.Count) { contexts.Keys.OrderBy(p => p).Take(contexts.Count - MaxCachedVerificationProcessCount + 1).ForEach(p => { if (contexts.TryRemove(p, out var value)) { value.Timer.CancelIfNotNull(); } }); }

@erikzhang How about this？

shargon

I think that the code it's ok

vncoelho · 2021-02-01T08:41:45Z

hi @ZhangTao1596, you are correct, I forgot to delete the OraclePlugin just for those clients, makes sense.
I did not test without it, but the analysis is precise.

src/StateService/Verification/VerificationService.cs

shargon · 2021-02-01T09:19:12Z

src/StateService/Verification/VerificationService.cs

+ {
+ Timeout = TimeSpan.FromMilliseconds(TimeoutMilliseconds),
+ };
+ var client = new RpcClient(http_client, url);


We should use ExtensiblePayload with p2p, but it could be done in a different PR

Yes. Using p2p, We won't need url and avoid inconsistence between urls and public keys.
I have code here https:/ZhangTao1596/neo-modules/tree/state-service-p2p.

@Tommo-L worries that use p2p will put more pressure on p2p. And he proposes add service discovery function.
Once we make agreement, I will create another pr.

If we use service discovery, will it expose the IP address?

Can we merge this for now. I will make p2p improvement pr later.

use p2p will put more pressure on p2p.

Yes

will it expose the IP address?

Maybe we can encrypt part of the service information so that only the relevant node can decrypt it, like:

service type + service information state: xxxxxxx oracle: yyyyyyy rpc: 192.168.1.1 ...

We can consider it later, after this pr.

@Tommo-L Can you create a issue of your idea. We can discuss there. Since oracle and state both need that.

Oracle can't use p2p message for siging!

We can discuss here #493

ProDog · 2021-02-01T10:33:50Z

Merge?

ZhangTao1596 added 3 commits January 18, 2021 18:33

add state validation

ca3fee1

register rpc

73cde36

Merge branch 'master' into state-service-validation

4c99958

ZhangTao1596 requested review from erikzhang and shargon January 19, 2021 01:57

superboyiii mentioned this pull request Jan 19, 2021

Preview5 Checklist neo-project/neo#2171

Closed

36 tasks

shargon reviewed Jan 19, 2021

View reviewed changes

src/StateService/Settings.cs Outdated Show resolved Hide resolved

src/StateService/StatePlugin.cs Outdated Show resolved Hide resolved

src/StateService/StatePlugin.cs Outdated Show resolved Hide resolved

shargon added 6 commits January 19, 2021 12:06

Clean condition

d0a9686

Use concurrent dictionary

71b80c4

lowercase

b183b04

Optimize

7b68864

Parallel foreach

4d35ea4

Clean enter

804d125

shargon reviewed Jan 19, 2021

View reviewed changes

src/StateService/Validation/ValidationProcess.cs Outdated Show resolved Hide resolved

vncoelho reviewed Jan 19, 2021

View reviewed changes

src/StateService/StatePlugin.cs Outdated Show resolved Hide resolved

ZhangTao1596 added 4 commits January 20, 2021 10:23

Merge branch 'master' into state-service-validation

39f9cc1

get wallet null check

a0202bf

fix some

8ca0710

rename validator to verifier

8a578dc

ZhangTao1596 requested review from vncoelho and shargon January 20, 2021 03:50

ZhangTao1596 changed the title ~~[StateService] Validation~~ [StateService] Verification Jan 20, 2021

ZhangTao1596 and others added 7 commits January 20, 2021 14:40

rename member

2bdbb38

optimize

bc33857

clean verification

568d330

optimize verification

f2371dd

verifier null check

911af71

check verifier when start

80b584c

Change string to Uri

70e7201

Qiao-Jin reviewed Jan 21, 2021

View reviewed changes

src/StateService/Verification/VerificationContext.cs Outdated Show resolved Hide resolved

erikzhang dismissed superboyiii’s stale review via a9e4dfb January 30, 2021 14:26

format

e83a9c3

shargon reviewed Jan 31, 2021

View reviewed changes

src/StateService/Verification/VerificationService.cs Outdated Show resolved Hide resolved

ZhangTao1596 added 2 commits January 31, 2021 18:19

fix some

cc8a4bb

concurrent

9c40c81

Merge branch 'master' into state-service-validation

f41366d

vncoelho reviewed Jan 31, 2021

View reviewed changes

src/StateService/Verification/VerificationService.cs Show resolved Hide resolved

Tommo-L reviewed Jan 31, 2021

View reviewed changes

src/StateService/StateService/config.json Outdated Show resolved Hide resolved

clear setting

e12ca40

shargon reviewed Feb 1, 2021

View reviewed changes

shargon previously approved these changes Feb 1, 2021

View reviewed changes

use list

7c85c11

ZhangTao1596 dismissed shargon’s stale review via 7c85c11 February 1, 2021 08:42

shargon requested changes Feb 1, 2021

View reviewed changes

src/StateService/Verification/VerificationService.cs Outdated Show resolved Hide resolved

Use ReadOnlySpan

23175b4

shargon reviewed Feb 1, 2021

View reviewed changes

src/StateService/Verification/VerificationService.cs Outdated Show resolved Hide resolved

optimize

02e6f95

shargon approved these changes Feb 1, 2021

View reviewed changes

vncoelho approved these changes Feb 1, 2021

View reviewed changes

erikzhang merged commit 1aac793 into neo-project:master Feb 1, 2021

ZhangTao1596 deleted the state-service-validation branch February 1, 2021 14:56

joeqian10 pushed a commit to joeqian10/neo-modules that referenced this pull request Apr 7, 2021

[StateService] Verification (neo-project#470)

63296f1

AnnaShaleva mentioned this pull request Nov 4, 2023

Store notification merkleRoot in TransactionState neo-project/neo#2940

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[StateService] Verification #470

[StateService] Verification #470

ZhangTao1596 commented Jan 18, 2021 •

edited

Loading

shargon Jan 31, 2021

ZhangTao1596 Jan 31, 2021

shargon Jan 31, 2021

ZhangTao1596 Jan 31, 2021 •

edited

Loading

ZhangTao1596 Jan 31, 2021

shargon Jan 31, 2021

ZhangTao1596 Jan 31, 2021

superboyiii commented Jan 31, 2021

ZhangTao1596 commented Jan 31, 2021

vncoelho left a comment •

edited

Loading

ZhangTao1596 commented Feb 1, 2021 •

edited

Loading

shargon Feb 1, 2021

erikzhang Feb 1, 2021

ZhangTao1596 Feb 1, 2021 •

edited

Loading

ZhangTao1596 Feb 1, 2021 •

edited

Loading

erikzhang Feb 1, 2021

ZhangTao1596 Feb 1, 2021

erikzhang Feb 1, 2021

ZhangTao1596 Feb 1, 2021

shargon left a comment

vncoelho commented Feb 1, 2021

shargon Feb 1, 2021

ZhangTao1596 Feb 1, 2021 •

edited

Loading

erikzhang Feb 1, 2021

ZhangTao1596 Feb 1, 2021

Tommo-L Feb 1, 2021

ZhangTao1596 Feb 1, 2021

Tommo-L Feb 1, 2021

Tommo-L Feb 1, 2021

ProDog commented Feb 1, 2021

	signatures.Add(index, sig);
	return true;
	return signatures.TryAdd(index, sig);

[StateService] Verification #470

[StateService] Verification #470

Conversation

ZhangTao1596 commented Jan 18, 2021 • edited Loading

Step 2/3 StateService

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZhangTao1596 Jan 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superboyiii commented Jan 31, 2021

ZhangTao1596 commented Jan 31, 2021

vncoelho left a comment • edited Loading

Choose a reason for hiding this comment

ZhangTao1596 commented Feb 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZhangTao1596 Feb 1, 2021 • edited Loading

Choose a reason for hiding this comment

ZhangTao1596 Feb 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shargon left a comment

Choose a reason for hiding this comment

vncoelho commented Feb 1, 2021

Choose a reason for hiding this comment

ZhangTao1596 Feb 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ProDog commented Feb 1, 2021

ZhangTao1596 commented Jan 18, 2021 •

edited

Loading

ZhangTao1596 Jan 31, 2021 •

edited

Loading

vncoelho left a comment •

edited

Loading

ZhangTao1596 commented Feb 1, 2021 •

edited

Loading

ZhangTao1596 Feb 1, 2021 •

edited

Loading

ZhangTao1596 Feb 1, 2021 •

edited

Loading

ZhangTao1596 Feb 1, 2021 •

edited

Loading