[Execution State] Memory usage and performance optimisation

[ramtinms]

This Epic captures some of the improvements to the execution state code (also known as Ledger) in terms of reducing memory and improving the speed of operations and proof verifications.

• Optimize atree to reduce Flow's mtrie size and reduce number of reads atree#292
• [Execution state] complete ledger - Separate storage of payloads  #1746
• [Execution state] Compact Batch Proofs #1749
• [Execution state] complete ledger - Separation of node types (leaf, intermediate)  #1745
• [Execution Node] Reduce memory use and speedup reading single register #2559
• Add Blob Cadence type to store metadata onchain cadence#1402
• Improving efficiency when storing composite values (specially nested ones) cadence#1854

[fxamacker]

Atree Inlining & Deduplication (Testnet Results)

Before/After Comparison Using Aug 14, 2024 Devnet State

Mtrie Nodes and Atree Payloads	Atree Payload Sizes (bytes)
Total Counts Before After Change mtrie nodes 1,182,195,945 343,949,052 -838,246,893 atree payloads 455,399,623 102,283,931 -353,115,692 Total Sizes Before After Change mtrie nodes 113.49 GB 33.02 GB -80.47 GB atree payloads 87.74 GB 69.28 GB -18.46 GB combined 201.23 GB 102.30 GB -98.93 GB	Before After count 455,399,623 102,283,931 mean 192.7 677.4 std 996.4 2,065.7 min 13 22 25% 88 199 50% 139 665 75% 206 1,043 99% 876 1,267 99.5% 1,078 1,453 max 5,853,892 5,853,892

NOTE: Testnet and mainnet results can differ, depending on usage, data, etc. Although testnet memory use was reduced by 1/2 on Aug 14 data, mainnet memory reduction (percentage wise) will likely be more modest.

See atree epic 292 for more details & charts:

• Optimize atree to reduce Flow's mtrie size and reduce number of reads atree#292
• https:/onflow/atree/releases/tag/v0.8.0-rc.5