v0.8.0

Atree v0.8.0 introduces payload inlining and metadata deduplication to improve memory use, storage size, and speed of various servers that use payloads (not limited to EN). This improves network stability and performance as we handle future data growth.

Inlining and deduplication reduces payload count and mtrie nodes. This reduces size and growth rate of execution state.

• Inlining stores small arrays and maps (previously stored in their own payload) into an existing payload.
• Deduplication removes redundant metadata (e.g. Cadence struct field names and type information) to reduce payload size.

This release passed multiple types of tests, including 1034 hours of smoke tests that concluded on August 19, 2024.

Results exceeded goals mentioned in Scaling Execution Node (forum announcement).

Note

Sept. 4, 2024: 🏆 Atree inlined 500 million payloads (-61%) and eliminated over 1 billion mtrie nodes on mainnet!

Impact of these payload and mtrie node reductions include:

• Reduced RAM use by hundreds of GB on each mainnet execution node.
• Reduced SSD storage use (e.g. checkpoint file sizes).
• Reduced network bandwidth for chunk data packs (mentioned by Peter).
• Speedup transactions by 5-8% (mentioned by Jan).
• Speedup network upgrades (e.g. Cadence 1.0 migrations, future migrations).
• Future efficiency of other servers, components, databases (files/cache/index), etc. that benefit from fewer payloads.

Sept. 30, 2024: 🏆 Beyond execution nodes, bluesign shared a 5x speedup in tinyAN bootstrap time!
"btw amazing work this atree inlining, my tinyAN bootstrap time improved like 5x"

Future: We can replace nonatree domain payloads with atree payloads to reduce mainnet payload count by another 20-30%.

While devnet (shown below) had better percentage improvements, mainnet eliminated more mtrie nodes (over 1 billion)!

Memory Reduction Before/After Atree Inlining & Deduplication (Devnet Aug 14, 2024)

Mtrie Nodes and Atree Payloads (Aug 14, 2024 Devnet)	Atree Payload Sizes (bytes)
Total Counts Before After Change mtrie nodes 1,182,195,945 343,949,052 -838,246,893 atree payloads 455,399,623 102,283,931 -353,115,692 Total Sizes Before After Change mtrie nodes 113.49 GB 33.02 GB -80.47 GB atree payloads 87.74 GB 69.28 GB -18.46 GB combined 201.23 GB 102.30 GB -98.93 GB	Before After count 455,399,623 102,283,931 mean 192.7 677.4 std 996.4 2,065.7 min 13 22 25% 88 199 50% 139 665 75% 206 1,043 99% 876 1,267 99.5% 1,078 1,453 max 5,853,892 5,853,892

What's Changed (since v0.6.0)

• Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
• Change Array.Get() to return Value instead of Storable by @fxamacker in #316
• Optimize max map value size to reduce number of registers by @fxamacker in #314
• Refactor to remove lint warnings by @fxamacker in #319
• Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
• Rename ID to ValueID by @fxamacker in #325
• Refactor creating new StorableSlab by @fxamacker in #324
• Unexport SlabID fields to prevent misuse by @fxamacker in #323
• Rename StorageID related types, vars, and funcs by @fxamacker in #322
• Add support for iterating only loaded values by @fxamacker in #311
• Refactor hasPointer for register inlining by @fxamacker in #327
• Reduce size of encoded array slab and bump version by @fxamacker in #330
• Add ignore filters to dependabot.yml by @fxamacker in #302
• Reduce encoded size of map and bump version by @fxamacker in #331
• Update README.md to describe Atree by @fxamacker in #332
• Update responsible disclosure link by @jribbink in #333
• Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
• Omit empty next slab ID in encoded map data slab by @fxamacker in #340
• Omit empty next slab ID in encoded array data slab by @fxamacker in #339
• Fix slab size when resetting mutable storable in Array by @fxamacker in #336
• Refactor encoding version and flag to add more flags by @fxamacker in #338
• Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
• Atree Register Inlining and Data Deduplication by @fxamacker in #342
• Update smoke test for atree inlining by @fxamacker in #348
• Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
• Make smoke tests check recently added data deduplication feature by @fxamacker in #350
• Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
• Add feature to support mutation for array and map iterators by @fxamacker in #359
• Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
• Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
• Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
• Use encoded type info to deduplicate extra data by @fxamacker in #381
• Fix error type for external errors during serialization by @fxamacker in #382
• Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
• Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
• Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
• Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
• Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404
• Check mutation of elements from readonly map iterator by @fxamacker in #410
• Check mutation of elements from readonly array iterator by @fxamacker in #411
• Update comment for NondeterministicFastCommit by @fxamacker in #412
• Update copyright notice to Flow Foundation by @fxamacker in #415
• Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
• Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428
• Merge feature/array-map-inlining (atree inlining feature branch) to main by @fxamacker in #429
• Add functions to check availablility of CBOR tag numbers by @fxamacker in #434
• Add SlabID.Address() by @fxamacker in #435
• Make smoke tests check if CBOR tag nums are reserved by @fxamacker in #436

Changes to CI and version bumps...

• Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in #289
• Bump actions/setup-go from 3 to 4 by @dependabot in #298
• Bump codecov/codecov-action from 3.1.1 to 3.1.3 by @dependabot in #300
• Bump codecov/codecov-action from 3.1.3 to 3.1.4 by @dependabot in #305
• Bump lukechampine.com/blake3 from 1.1.7 to 1.2.1 by @dependabot in #308
• Bump github.com/stretchr/testify from 1.8.2 to 1.8.4 by @dependabot in #309
• Bump safer-golangci-lint to 1.52.2 by @fxamacker in #304
• Bump codecov/codecov-action from 3.1.4 to 4.0.2 by @dependabot in #368
• Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #371
• Pin CodeQL workflow dependencies by @fxamacker in #363
• Pin dependencies in coverage.yml by @fxamacker in #3...

v0.8.0-rc.6

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since 0.8.0-rc.5

To reduce risks, new functions were added to help projects using Atree avoid producing undecodable data in storage.

• Add IsCBORTagNumberRangeAvailable() and ReservedCBORTagNumberRange() to check availablility of CBOR tag numbers by @fxamacker in #434

• Add SlabID.Address() by @fxamacker in #435

Changes to CI and version bumps...

• Bump actions/setup-go from 4.1.0 to 5.0.2 by @dependabot in #427
• Bump github/codeql-action from 3.25.13 to 3.25.15 by @dependabot in #430
• Bump github/codeql-action from 3.25.15 to 3.26.0 by @dependabot in #432

Full Changelog: v0.8.0-rc.5...v0.8.0-rc.6

v0.8.0-rc.5

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.6.0 (main branch)

v0.8.0-rc.5 merges atree inlining feature branch to main branch after extensive testing.

• Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
• Change Array.Get() to return Value instead of Storable by @fxamacker in #316
• Optimize max map value size to reduce number of registers by @fxamacker in #314
• Refactor to remove lint warnings by @fxamacker in #319
• Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
• Rename ID to ValueID by @fxamacker in #325
• Refactor creating new StorableSlab by @fxamacker in #324
• Unexport SlabID fields to prevent misuse by @fxamacker in #323
• Rename StorageID related types, vars, and funcs by @fxamacker in #322
• Add support for iterating only loaded values by @fxamacker in #311
• Refactor hasPointer for register inlining by @fxamacker in #327
• Reduce size of encoded array slab and bump version by @fxamacker in #330
• Add ignore filters to dependabot.yml by @fxamacker in #302
• Reduce encoded size of map and bump version by @fxamacker in #331
• Update README.md to describe Atree by @fxamacker in #332
• Update responsible disclosure link by @jribbink in #333
• Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
• Omit empty next slab ID in encoded map data slab by @fxamacker in #340
• Omit empty next slab ID in encoded array data slab by @fxamacker in #339
• Fix slab size when resetting mutable storable in Array by @fxamacker in #336
• Refactor encoding version and flag to add more flags by @fxamacker in #338
• Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
• Atree Register Inlining and Data Deduplication by @fxamacker in #342
• Update smoke test for atree inlining by @fxamacker in #348
• Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
• Make smoke tests check recently added data deduplication feature by @fxamacker in #350
• Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
• Add feature to support mutation for array and map iterators by @fxamacker in #359
• Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
• Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
• Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
• Use encoded type info to deduplicate extra data by @fxamacker in #381
• Fix error type for external errors during serialization by @fxamacker in #382
• Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
• Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
• Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396
• Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
• Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404
• Check mutation of elements from readonly map iterator by @fxamacker in #410
• Check mutation of elements from readonly array iterator by @fxamacker in #411
• Update comment for NondeterministicFastCommit by @fxamacker in #412
• Update copyright notice to Flow Foundation by @fxamacker in #415
• Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
• Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428
• Merge feature/array-map-inlining (atree inlining feature branch) to main by @fxamacker in #429

Changes to CI and version bumps...

• Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in #289
• Bump actions/setup-go from 3 to 4 by @dependabot in #298
• Bump codecov/codecov-action from 3.1.1 to 3.1.3 by @dependabot in #300
• Bump codecov/codecov-action from 3.1.3 to 3.1.4 by @dependabot in #305
• Bump lukechampine.com/blake3 from 1.1.7 to 1.2.1 by @dependabot in #308
• Bump github.com/stretchr/testify from 1.8.2 to 1.8.4 by @dependabot in #309
• Bump safer-golangci-lint to 1.52.2 by @fxamacker in #304
• Bump codecov/codecov-action from 3.1.4 to 4.0.2 by @dependabot in #368
• Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #371
• Pin CodeQL workflow dependencies by @fxamacker in #363
• Pin dependencies in coverage.yml by @fxamacker in #366
• Pin dependencies in ci.yml by @fxamacker in #367
• Bump github/codeql-action from 3.24.6 to 3.24.8 by @dependabot in #378
• Bump github/codeql-action from 3.24.8 to 3.24.9 by @dependabot in #379
• Bump github/codeql-action from 3.24.9 to 3.24.10 by @dependabot in #384
• Bump codecov/codecov-action from 4.0.1 to 4.2.0 by @dependabot in #385
• Bump lukechampine.com/blake3 from 1.2.1 to 1.2.2 by @dependabot in #383
• Bump github/codeql-action from 3.24.10 to 3.25.0 by @dependabot in #389
• Bump github/codeql-action from 3.25.0 to 3.25.1 by @dependabot in #393
• Bump codecov/codecov-action from 4.2.0 to 4.3.0 by @dependabot in #390
• Bump github/codeql-action from 3.25.1 to 3.25.3 by @dependabot in #400
• Bump codecov/codecov-action from 4.3.0 to 4.3.1 by @dependabot in #401
• Bump github/codeql-action from 3.25.3 to 3.25.5 by @dependabot in #405
• Bump codecov/codecov-action from 4.3.1 to 4.4.1 by @dependabot in #408
• Bump github/codeql-action from 3.25.5 to 3.25.6 by @dependabot in #413
• Bump lukechampine.com/blake3 from 1.2.2 to 1.3.0 by @dependabot in #402
• Bump github/codeql-action from 3.25.6 to 3.25.7 by @dependabot in #414
• Bump github/codeql-action from 3.25.7 to 3.25.8 by @dependabot in #418
• Bump github/codeql-action from 3.25.8 to 3.25.10 by @dependabot in #420
• Bump codecov/codecov-action from 4.4.1 to 4.5.0 by @dependabot in #419
• Bump github/codeql-action from 3.25.10 to 3.25.11 by @dependabot in #423
• Bump golangci-lint to 1.56.2 by @fxamacker in #422
• Bump github/codeql-action from 3.25.11 to 3.25.12 by @dependabot in #424
• Bump actions/checkout from 4.1.1 to 4.1.7 by @dependabot in #425
• Bump github/codeql-action from 3.25.12 to 3.25.13 by @dependabot in #426

New Contributors

• @jribbink made their first contribution in #333

Full Changelog: v0.6.0...v0.8.0-rc.5

v0.8.0-rc.4

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.8.0-rc.3

Changes in v0.8.0-rc.4 are limited to updated copyright notice, comments, and dependency version bumps.

More details...

• Bump codecov/codecov-action from 3.1.4 to 4.0.2 by @dependabot in #368
• Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by @dependabot in #371
• Pin CodeQL workflow dependencies by @fxamacker in #363
• Pin dependencies in coverage.yml by @fxamacker in #366
• Pin dependencies in ci.yml by @fxamacker in #367
• Bump github/codeql-action from 3.24.6 to 3.24.8 by @dependabot in #378
• Bump github/codeql-action from 3.24.8 to 3.24.9 by @dependabot in #379
• Bump github/codeql-action from 3.24.9 to 3.24.10 by @dependabot in #384
• Bump codecov/codecov-action from 4.0.1 to 4.2.0 by @dependabot in #385
• Bump lukechampine.com/blake3 from 1.2.1 to 1.2.2 by @dependabot in #383
• Bump github/codeql-action from 3.24.10 to 3.25.0 by @dependabot in #389
• Bump github/codeql-action from 3.25.0 to 3.25.1 by @dependabot in #393
• Bump codecov/codecov-action from 4.2.0 to 4.3.0 by @dependabot in #390
• Bump github/codeql-action from 3.25.1 to 3.25.3 by @dependabot in #400
• Bump codecov/codecov-action from 4.3.0 to 4.3.1 by @dependabot in #401
• Bump github/codeql-action from 3.25.3 to 3.25.5 by @dependabot in #405
• Bump codecov/codecov-action from 4.3.1 to 4.4.1 by @dependabot in #408
• Update comment for NondeterministicFastCommit by @fxamacker in #412
• Bump github/codeql-action from 3.25.5 to 3.25.6 by @dependabot in #413
• Bump lukechampine.com/blake3 from 1.2.2 to 1.3.0 by @dependabot in #402
• Bump github/codeql-action from 3.25.6 to 3.25.7 by @dependabot in #414
• Bump github/codeql-action from 3.25.7 to 3.25.8 by @dependabot in #418
• Update copyright notice to Flow Foundation by @fxamacker in #415
• Update copyright notice to Flow Foundation (in main branch) by @fxamacker in #417
• Bump github/codeql-action from 3.25.8 to 3.25.10 by @dependabot in #420
• Bump codecov/codecov-action from 4.4.1 to 4.5.0 by @dependabot in #419
• Bump github/codeql-action from 3.25.10 to 3.25.11 by @dependabot in #423
• Bump golangci-lint to 1.56.2 by @fxamacker in #422
• Bump github/codeql-action from 3.25.11 to 3.25.12 by @dependabot in #424
• Bump actions/checkout from 4.1.1 to 4.1.7 by @dependabot in #425
• Bump github/codeql-action from 3.25.12 to 3.25.13 by @dependabot in #426
• Update feature/array-map-inlining (atree inlining feature branch) by @fxamacker in #428

Full Changelog: v0.8.0-rc.3...v0.8.0-rc.4

v0.7.0-rc.3

The v0.7.x does not include atree inlining. Atree inlining is introduced in v0.8.0-rc.1 with breaking changes to data format.

What's Changed Since v0.7.0-rc.2

• Update copyright notice to Flow Foundation (in feature/stable-cadence) by @fxamacker in #416

Full Changelog: v0.7.0-rc.2...v0.7.0-rc.3

v0.8.0-rc.3

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.8.0-rc.2

• Check mutation of elements from readonly map iterator by @fxamacker in #410
• Check mutation of elements from readonly array iterator by @fxamacker in #411

Full Changelog: v0.8.0-rc.2...v0.8.0-rc.3

v0.8.0-rc.2

Release 0.8.x (since v0.8.0-rc.1) introduces atree inlining and deduplication, so the data format is a breaking change from 0.7.x.

What's Changed Since v0.8.0-rc.1

• Add NonderterministicFastCommit to speed up migrations when ordering isn't required by @fxamacker in #403
• Add BatchPreload to decode slabs in parallel and cache by @fxamacker in #404

These new features are intended to be used by state migration programs in onflow/flow-go to speedup migrations.

Full Changelog: v0.8.0-rc.1...v0.8.0-rc.2

v0.7.0-rc.2

The v0.7.x does not include atree inlining. Atree inlining is introduced in v0.8.0-rc.1 with breaking changes to data format.

What's Changed Since v0.7.0-rc.1

• Add NonderterministicFastCommit to speed up migrations (for branch without atree inlining) by @fxamacker in #406
• Add BatchPreload to decode slabs in parallel and cache (for branch without atree inlining) by @fxamacker in #407

These new features are intended to be used by state migration programs in onflow/flow-go to speedup migrations.

Full Changelog: v0.7.0-rc.1...v0.7.0-rc.2

v0.8.0-rc.1

Release 0.8.0-rc.1 introduces atree inlining and deduplication, so the data format is a breaking change.

Atree inlining and deduplication is primarily in these PRs:

• Atree Register Inlining and Data Deduplication in #342
• Add readonly iterators and support value mutations only from non-readonly iterators in #345
• Reduce RAM and persistent storage by deduplicating inlined dict type info in #369

NOTE: Atree Register Inlining Project includes other work: migration (in onflow/flow-go) and integration (in onflow/cadence).

What's Changed

• Change OrderedMap.Get() to return Value instead of Storable by @fxamacker in #318
• Change Array.Get() to return Value instead of Storable by @fxamacker in #316
• Optimize max map value size to reduce number of registers by @fxamacker in #314
• Refactor to remove lint warnings by @fxamacker in #319
• Add Array.ID() and OrderedMap.ID() by @fxamacker in #321
• Rename ID to ValueID by @fxamacker in #325
• Refactor creating new StorableSlab by @fxamacker in #324
• Unexport SlabID fields to prevent misuse by @fxamacker in #323
• Rename StorageID related types, vars, and funcs by @fxamacker in #322
• Add support for iterating only loaded values by @fxamacker in #311
• Refactor hasPointer for register inlining by @fxamacker in #327
• Reduce size of encoded array slab and bump version by @fxamacker in #330
• Reduce encoded size of map and bump version by @fxamacker in #331
• Update README.md to describe Atree by @fxamacker in #332
• Update responsible disclosure link by @jribbink in #333
• Fix slab size when resetting mutable storable in OrderedMap by @fxamacker in #337
• Omit empty next slab ID in encoded map data slab by @fxamacker in #340
• Omit empty next slab ID in encoded array data slab by @fxamacker in #339
• Fix slab size when resetting mutable storable in Array by @fxamacker in #336
• Refactor encoding version and flag to add more flags by @fxamacker in #338
• Add readonly iterators and support value mutations only from non-readonly iterators by @fxamacker in #345
• Atree Register Inlining and Data Deduplication by @fxamacker in #342
• Update smoke test for atree inlining by @fxamacker in #348
• Update for Cadence integration for atree inlining and deduplication by @fxamacker in #352
• Make smoke tests check recently added data deduplication feature by @fxamacker in #350
• Remove ContainerStorable.EncodeAsElement by @fxamacker in #354
• Add feature to support mutation for array and map iterators by @fxamacker in #359
• Add support for changing type info of atree arrays (atree inlining branch) by @fxamacker in #376
• Add support for changing type info of atree maps (atree inlining branch) by @fxamacker in #377
• Reduce RAM and persistent storage by deduplicating inlined dict type info by @fxamacker in #369
• Use encoded type info to deduplicate extra data by @fxamacker in #381
• Fix error type for external errors during serialization by @fxamacker in #382
• Add PersistentSlabStorage.GetAllChildReferences() for atree inlining by @fxamacker in #392
• Add feature to enable atree inlining migration to fix references to non-existent registers by @fxamacker in #388
• Fix migration filter for old unreferenced slabs (atree inlining feature branch) by @fxamacker in #396

CI / Dependabot

• Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in #289
• Bump actions/setup-go from 3 to 4 by @dependabot in #298
• Bump codecov/codecov-action from 3.1.1 to 3.1.3 by @dependabot in #300
• Add ignore filters to dependabot.yml by @fxamacker in #302
• Bump safer-golangci-lint to 1.52.2 by @fxamacker in #304
• Bump codecov/codecov-action from 3.1.3 to 3.1.4 by @dependabot in #305
• Bump lukechampine.com/blake3 from 1.1.7 to 1.2.1 by @dependabot in #308
• Bump github.com/stretchr/testify from 1.8.2 to 1.8.4 by @dependabot in #309

New Contributors

• @jribbink made their first contribution in #333

Full Changelog: v0.6.0...v0.8.0-rc.1

v0.7.0-rc.1

The v0.7.0-rc.1 does not include atree inlining. Atree inlining is introduced in v0.8.0-rc.1 with breaking changes to data format.

What's Changed

• Add Array.SetType() to allow updating TypeInfo by @fxamacker in #374
• Add OrderedMap.SetType() to allow updating TypeInfo by @fxamacker in #375
• Add PersistentSlabStorage.GetAllChildReferences() by @fxamacker in #391
• Add feature to enable migrations to fix references to non-existent registers by @fxamacker in #387
• Fix SlabIterator to include nested storage ID (non-inlining feature branch) by @fxamacker in #398
• Fix migration filter for old unreferenced slabs (non-inlining feature branch) by @fxamacker in #399

CI / Dependabot

• Bump github.com/stretchr/testify from 1.8.1 to 1.8.2 by @dependabot in #289
• Bump actions/setup-go from 3 to 4 by @dependabot in #298
• Bump codecov/codecov-action from 3.1.1 to 3.1.3 by @dependabot in #300

Full Changelog: v0.6.0...v0.7.0-rc.1