[db] add KvVersioned interface #4041
Conversation
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4041 +/- ##
==========================================
+ Coverage 76.51% 77.01% +0.50%
==========================================
Files 340 342 +2
Lines 29273 29369 +96
==========================================
+ Hits 22397 22618 +221
+ Misses 5761 5654 -107
+ Partials 1115 1097 -18 ☔ View full report in Codecov by Sentry. |
db/db_versioned.go
Outdated
| } | ||
|
|
||
| // SetVersion sets the version which a Get()/Put() wants to access | ||
| func (b *BoltDBVersioned) SetVersion(v uint64) KvVersioned { |
There was a problem hiding this comment.
the v means blockHeight in production?
db/db_versioned.go
Outdated
| } | ||
|
|
||
| // CreateVersionedNamespace creates a namespace to store versioned keys | ||
| func (b *BoltDBVersioned) CreateVersionedNamespace(ns string, n uint32) error { |
There was a problem hiding this comment.
n is the size of key in this namespace
db/db_versioned.go
Outdated
| BoltDBVersioned struct { | ||
| *BoltDB | ||
| version uint64 // version for Get/Put() | ||
| versioned map[string]int // buckets for versioned keys |
There was a problem hiding this comment.
If there are many keys, will these two maps be very large?
There was a problem hiding this comment.
i think the map key should be namespace instead of key, is it right?
There was a problem hiding this comment.
yes, the map key is the namespace
| // 1. "mta" -- regular bucket storing metadata, key is not versioned | ||
| // 2. "act" -- versioned namespace, key length = 20 | ||
| // 3. "stk" -- versioned namespace, key length = 8 | ||
| KvVersioned interface { |
There was a problem hiding this comment.
I prefer the following interface:
KvVersioned interface{
Version(ns string, key []byte) (ver uint64, err error)
WithVersion(ver uint64) KVStore
CreateNamespace(ns string, keySize uint32) error
}Simplify the definition of KvVersioned, it does not support operations such as put/get itself, and must specify the version before proceeding. If need an interface that supports both versioned and non-versioned operations, can define another one.
There was a problem hiding this comment.
If this is an interface that supports both versioned and non-versioned operations at the same time, a method IsVersionedNamespaces(ns string) should also be added, so that when the interface is used, it is known whether to call Put() or SetVersion().Put().
Or the Put() is adapted for both versioned and non-versioned:
- if versioned, it means put to latest version
- if non-versioned, just put
There was a problem hiding this comment.
good suggestions, updated PR
yes, Put() is a common interface for both versioned and non-versioned, the code will check inside and process accordingly
dustinxie
changed the title
db/db_versioned.go
Outdated
| // Put writes a <key, value> record | ||
| func (b *BoltDBVersioned) Put(ns string, key, value []byte) error { | ||
| if !b.IsReady() { | ||
| return ErrDBNotStarted | ||
| } | ||
| // TODO: implement Put | ||
| return nil | ||
| } | ||
|
|
||
| // Get retrieves the most recent version | ||
| func (b *BoltDBVersioned) Get(ns string, key []byte) ([]byte, error) { | ||
| if !b.IsReady() { | ||
| return nil, ErrDBNotStarted | ||
| } | ||
| // TODO: implement Get | ||
| return nil, nil | ||
| } | ||
|
|
||
| // Delete deletes a record,if key is nil,this will delete the whole bucket | ||
| func (b *BoltDBVersioned) Delete(ns string, key []byte) error { | ||
| if !b.IsReady() { | ||
| return ErrDBNotStarted | ||
| } | ||
| // TODO: implement Delete | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Do we still need Put, Get, Delete the three methods?
There was a problem hiding this comment.
yes, that is the core function of this KvVersioned concept, and will be implemented in later PRs
| } | ||
|
|
||
| // Delete deletes a record,if key is nil,this will delete the whole bucket | ||
| func (b *BoltDBVersioned) Delete(ns string, key []byte) error { |
db/db_versioned.go
Outdated
| // Version returns the key's most recent version | ||
| Version(string, []byte) (uint64, error) | ||
| // KvStore returns a handler to perform versioned read/write | ||
| KvStore() KvVersioned |
There was a problem hiding this comment.
KvStore(version uint64) KvVersioned
|
pls add more details as context in the description |
| uint32 keyLen = 2; | ||
| } | ||
|
|
||
| message KeyMeta { |
There was a problem hiding this comment.
will be used in future PR
|
From my understand, the idea of pr is to build a mapping for deduplicated data. Changing the implementation of KVStore is more reasonable than changing boltdb itself, where does simple kv storage jobs. |
it's not changing boltdb, rather it's using boltdb's underlying API to construct a KVStore that can return multiple versions of value for the same key |
|
Description
Currently the staking index DB saves all delegates and candidates at each epoch, the DB size grows very rapidly due to large size of data stored.
This PR introduces a new DB which can save an item at different version (blockchain's height) only when the value changes. In a quick POC testing, this has significantly reduced the DB size from ~80GB to ~4GB
Fixes #(issue)
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Test Configuration:
Checklist: