General Message Queue Pallet (paritytech#12485)

* The message queue * Make fully generic * Refactor * Docs * Refactor * Use iter not slice * Per-origin queues * Multi-queue processing * Introduce MaxReady * Remove MaxReady in favour of ready ring * Cleanups * ReadyRing and tests * Stale page reaping * from_components -> from_parts Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Move WeightCounter to sp_weights Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add MockedWeightInfo Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Deploy to kitchensink Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Use WeightCounter Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Small fixes and logging Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add service_page Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Typo Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Move service_page below service_queue Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add service_message Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Use correct weight function Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Overweight execution * Refactor * Missing file * Fix WeightCounter usage in scheduler Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix peek_index Take into account that decoding from a mutable slice modifies it. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add tests and bench service_page_item Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add debug_info Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add no-progress check to service_queues Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add more benches Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Bound from_message and try_append_message Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add PageReaped event Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Rename BookStateOf and BookStateFor Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Update tests and remove logging Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Remove redundant per-message origins; add footprint() and sweep_queue() * Move testing stuff to mock.rs Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add integration test Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix no-progress check Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix debug_info Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fixup merge and tests Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix footprint tracking * Introduce * Formatting * OverweightEnqueued event, auto-servicing config item * Update tests and benchmarks Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Clippy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add tests Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Provide change handler * Add missing BookStateFor::insert and call QueueChangeHandler Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Docs Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Update benchmarks and weights Signed-off-by: Oliver Tale-Yazdi <[email protected]> * More tests... Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Use weight metering functions Signed-off-by: Oliver Tale-Yazdi <[email protected]> * weightInfo::process_message_payload is gone Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add defensive_saturating_accrue Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Rename WeightCounter to WeightMeter Ctr+Shift+H should do the trick. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Test on_initialize Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add module docs Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Remove origin from MaxMessageLen The message origin is not encoded into the heap and does therefore not influence the max message length anymore. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add BoundedVec::as_slice Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Test Page::{from_message, try_append_message} Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fixup docs Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Docs * Do nothing in sweep_queue if the queue does not exist ... otherwise it inserts default values into the storage. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Test ring (un)knitting Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Upgrade stress-test Change the test to not assume that all queued messages will be processed in the next block but split it over multiple. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * More tests... Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <[email protected]> * clippy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Rename BoundedVec::as_slice to as_bounded_slice Conflicts with deref().as_slice() otherwise. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix imports Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Remove ReadyRing struct Was used for testing only. Instead use 'fn assert_ring' which also check the service head and backlinks. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix stale page watermark Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Cleanup Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix test feature and clippy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * QueueChanged handler is called correctly Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Update benches Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Abstract testing functions Signed-off-by: Oliver Tale-Yazdi <[email protected]> * More tests Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Cleanup Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Clippy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * fmt Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Simplify tests Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Make stuff compile Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Extend overweight execution benchmark Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Remove TODOs Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Test service queue with faulty MessageProcessor Signed-off-by: Oliver Tale-Yazdi <[email protected]> * fmt Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Update pallet ui tests to 1.65 Signed-off-by: Oliver Tale-Yazdi <[email protected]> * More docs Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Review doc fixes Co-authored-by: Robert Klotzner <[email protected]> Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add weight_limit to extrinsic weight of execute_overweight * Correctly return unused weight * Return actual weight consumed in do_execute_overweight * Review fixes Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Set version 7.0.0-dev Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Make it compile Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Switch message_size to u64 Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Switch message_count to u64 Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix benchmarks Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Make CI green Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Docs * Update tests Signed-off-by: Oliver Tale-Yazdi <[email protected]> * ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue * Dont mention README.md in the Cargo.toml Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Remove reference to readme Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: parity-processbot <> Co-authored-by: Robert Klotzner <[email protected]> Co-authored-by: Keith Yeung <[email protected]>
gear-tech · Feb 27, 2023 · dcbd710 · dcbd710
1 parent 5b9b5b8
commit dcbd710
Show file tree

Hide file tree

Showing 20 changed files with 3,883 additions and 15 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -121,6 +121,7 @@ members = [
  "frame/offences/benchmarking",
  "frame/preimage",
  "frame/proxy",
+ "frame/message-queue",
  "frame/nomination-pools",
  "frame/nomination-pools/fuzzer",
  "frame/nomination-pools/benchmarking",

diff --git a/bin/node/runtime/Cargo.toml b/bin/node/runtime/Cargo.toml
@@ -75,6 +75,7 @@ pallet-indices = { version = "4.0.0-dev", default-features = false, path = "../.
 pallet-identity = { version = "4.0.0-dev", default-features = false, path = "../../../frame/identity" }
 pallet-lottery = { version = "4.0.0-dev", default-features = false, path = "../../../frame/lottery" }
 pallet-membership = { version = "4.0.0-dev", default-features = false, path = "../../../frame/membership" }
+pallet-message-queue = { version = "7.0.0-dev", default-features = false, path = "../../../frame/message-queue" }
 pallet-mmr = { version = "4.0.0-dev", default-features = false, path = "../../../frame/merkle-mountain-range" }
 pallet-multisig = { version = "4.0.0-dev", default-features = false, path = "../../../frame/multisig" }
 pallet-nomination-pools = { version = "1.0.0", default-features = false, path = "../../../frame/nomination-pools"}
@@ -150,6 +151,7 @@ std = [
  "sp-inherents/std",
  "pallet-lottery/std",
  "pallet-membership/std",
+ "pallet-message-queue/std",
  "pallet-mmr/std",
  "pallet-multisig/std",
  "pallet-nomination-pools/std",
@@ -229,6 +231,7 @@ runtime-benchmarks = [
  "pallet-indices/runtime-benchmarks",
  "pallet-lottery/runtime-benchmarks",
  "pallet-membership/runtime-benchmarks",
+ "pallet-message-queue/runtime-benchmarks",
  "pallet-mmr/runtime-benchmarks",
  "pallet-multisig/runtime-benchmarks",
  "pallet-nomination-pools-benchmarking/runtime-benchmarks",
@@ -282,6 +285,7 @@ try-runtime = [
  "pallet-identity/try-runtime",
  "pallet-lottery/try-runtime",
  "pallet-membership/try-runtime",
+ "pallet-message-queue/try-runtime",
  "pallet-mmr/try-runtime",
  "pallet-multisig/try-runtime",
  "pallet-nomination-pools/try-runtime",

diff --git a/bin/node/runtime/src/lib.rs b/bin/node/runtime/src/lib.rs
@@ -1135,6 +1135,25 @@ impl pallet_bounties::Config for Runtime {
  type ChildBountyManager = ChildBounties;
 }
 
+parameter_types! {
+ /// Allocate at most 20% of each block for message processing.
+ ///
+ /// Is set to 20% since the scheduler can already consume a maximum of 80%.
+ pub MessageQueueServiceWeight: Option<Weight> = Some(Perbill::from_percent(20) * RuntimeBlockWeights::get().max_block);
+}
+
+impl pallet_message_queue::Config for Runtime {
+ type RuntimeEvent = RuntimeEvent;
+ type WeightInfo = ();
+ /// NOTE: Always set this to `NoopMessageProcessor` for benchmarking.
+ type MessageProcessor = pallet_message_queue::mock_helpers::NoopMessageProcessor;
+ type Size = u32;
+ type QueueChangeHandler = ();
+ type HeapSize = ConstU32<{ 64 * 1024 }>;
+ type MaxStale = ConstU32<128>;
+ type ServiceWeight = MessageQueueServiceWeight;
+}
+
 parameter_types! {
  pub const ChildBountyValueMinimum: Balance = 1 * DOLLARS;
 }
@@ -1699,6 +1718,7 @@ construct_runtime!(
  RankedPolls: pallet_referenda::<Instance2>,
  RankedCollective: pallet_ranked_collective,
  FastUnstake: pallet_fast_unstake,
+ MessageQueue: pallet_message_queue,
  }
 );
 
@@ -1793,6 +1813,7 @@ mod benches {
  [pallet_indices, Indices]
  [pallet_lottery, Lottery]
  [pallet_membership, TechnicalMembership]
+ [pallet_message_queue, MessageQueue]
  [pallet_mmr, Mmr]
  [pallet_multisig, Multisig]
  [pallet_nomination_pools, NominationPoolsBench::<Runtime>]

diff --git a/frame/message-queue/Cargo.toml b/frame/message-queue/Cargo.toml
@@ -0,0 +1,53 @@
+[package]
+authors = ["Parity Technologies <[email protected]>"]
+edition = "2021"
+name = "pallet-message-queue"
+version = "7.0.0-dev"
+license = "Apache-2.0"
+homepage = "https://substrate.io"
+repository = "https:/paritytech/substrate/"
+description = "FRAME pallet to queue and process messages"
+
+[dependencies]
+codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false, features = ["derive"] }
+scale-info = { version = "2.1.2", default-features = false, features = ["derive"] }
+serde = { version = "1.0.137", optional = true, features = ["derive"] }
+log = { version = "0.4.17", default-features = false }
+
+sp-core = { version = "7.0.0", default-features = false, path = "../../primitives/core" }
+sp-io = { version = "7.0.0", default-features = false, path = "../../primitives/io" }
+sp-runtime = { version = "7.0.0", default-features = false, path = "../../primitives/runtime" }
+sp-std = { version = "5.0.0", default-features = false, path = "../../primitives/std" }
+sp-arithmetic = { version = "6.0.0", default-features = false, path = "../../primitives/arithmetic" }
+sp-weights = { version = "4.0.0", default-features = false, path = "../../primitives/weights" }
+
+frame-benchmarking = { version = "4.0.0-dev", default-features = false, optional = true, path = "../benchmarking" }
+frame-support = { version = "4.0.0-dev", default-features = false, path = "../support" }
+frame-system = { version = "4.0.0-dev", default-features = false, path = "../system" }
+
+[dev-dependencies]
+sp-tracing = { version = "6.0.0", path = "../../primitives/tracing" }
+rand = "0.8.5"
+rand_distr = "0.4.3"
+
+[features]
+default = ["std"]
+std = [
+ "codec/std",
+ "scale-info/std",
+ "sp-core/std",
+ "sp-io/std",
+ "sp-runtime/std",
+ "sp-std/std",
+ "sp-arithmetic/std",
+ "sp-weights/std",
+ "frame-benchmarking?/std",
+ "frame-support/std",
+ "frame-system/std",
+]
+runtime-benchmarks = [
+ "frame-benchmarking/runtime-benchmarks",
+ "frame-support/runtime-benchmarks",
+ "frame-system/runtime-benchmarks",
+]
+try-runtime = ["frame-support/try-runtime"]