Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KV + FST backup system #5

Closed
valeriansaliou opened this issue Mar 6, 2019 · 4 comments
Closed

KV + FST backup system #5

valeriansaliou opened this issue Mar 6, 2019 · 4 comments
Labels
feature Anything that is related to a new feature
Milestone

Comments

@valeriansaliou
Copy link
Owner

No description provided.

@valeriansaliou valeriansaliou self-assigned this Mar 6, 2019
@valeriansaliou valeriansaliou modified the milestones: 1.0.0, RUSH-11_03_2019, RUSH-15_03_2019 Mar 6, 2019
@valeriansaliou valeriansaliou modified the milestones: RUSH-17_03_2019, RUSH-15_03_2019 Mar 13, 2019
@valeriansaliou valeriansaliou added the enhancement Enhancement to an existing feature label Mar 16, 2019
@valeriansaliou valeriansaliou changed the title RocksDB + FST backup system (over channel protocol; ie. control mode) KV + FST backup system (over channel protocol; ie. control mode) Mar 16, 2019
@valeriansaliou
Copy link
Owner Author

valeriansaliou commented Mar 16, 2019

  • KV: Stream the RocksDB database to a text file, with each binary key as [u8 + u32] numbers and contents as a stream of u32 numbers; do this per-(collection, bucket);
  • FST: Stream the FST words to an ordered text file on-disk; this way we can re-import later on to a newer-version FST library (as the FST library may break the on-disk format between versions, this will let us write migration scripts easily); do this per-(collection, bucket);

@valeriansaliou valeriansaliou changed the title KV + FST backup system (over channel protocol; ie. control mode) KV + FST backup system Mar 17, 2019
@valeriansaliou
Copy link
Owner Author

Do this over channel protocol; ie. control mode.

@valeriansaliou
Copy link
Owner Author

Also add an option to restore a backup (per-collection or per-(collection, bucket) from given path) from the channel in control mode. This would over-write any existing database if needed (KV + FST).

@valeriansaliou
Copy link
Owner Author

Benefits of a text-based backup file format is that we generate human-readable backups that can be used for debugging, and that can also be used for easily converting a Sonic binary DB from an older format to a newer format, by bridging via the backup text format.

@valeriansaliou valeriansaliou removed their assignment Mar 17, 2019
@valeriansaliou valeriansaliou added feature Anything that is related to a new feature and removed enhancement Enhancement to an existing feature labels Mar 20, 2019
@valeriansaliou valeriansaliou added this to the v1.2.0 milestone Mar 23, 2019
valeriansaliou added a commit that referenced this issue Apr 16, 2019
Signed-off-by: Valerian Saliou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Anything that is related to a new feature
Projects
None yet
Development

No branches or pull requests

1 participant