-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ENH] Use foyer for block cache #2431
Conversation
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @Ishiihara and the rest of your teammates on Graphite |
aaab13c
to
b253dcc
Compare
b253dcc
to
ce5123c
Compare
for i in 0..n { | ||
let key = format!("key{}", i); | ||
let read = block.get::<&str, &str>("prefix", &key); | ||
values_before_flush.push(read.unwrap().to_string()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit - would prefer if the test here collected the values we generate instead of reading them from the block. avoid bugs where we persist the value wrong but its consistent
@@ -573,6 +629,14 @@ mod tests { | |||
blockfile_provider: | |||
Arrow: | |||
max_block_size_bytes: 16384 | |||
block_manager_config: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note - we will have to remember to update the config in staging when we merge this otherwise staging will crash
str::FromStr, | ||
sync::{atomic::AtomicU32, Arc}, | ||
}; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit - no whitespace between use
*Summarize the changes made by this PR.* - Improvements & Bug fixes - Makes the block size configurable, leaving default the same as before. - Add Blockfile provider config, removing a todo - Runs config tests in serial, since they can race - New functionality - None really *How are these changes tested?* Existing tests, this is a non-functional change - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust None
*Summarize the changes made by this PR.* - Improvements & Bug fixes - Fixes allowed_ids and disallowed_ids to also take care of updates/deletes/upserts. For e.g. if there is an update on the log that does not update the embedding and it is in the query list then today we are never going to return this record even if it is in the top k - Adds sync points to test_embeddings + increase test timeout - Adds another rule in test_embeddings for compaction - Suppresses health check warning for filtering too much - Fixes the case when trying to commit and flush an empty block (can happen due to deletes). Sparse index start key can also get changed to something that is not SparseIndexDelimiter::Start. We decided to go ahead with flushing a dummy block if blockfile becomes fully empty so that our segment abstraction is only uninitialized until the first compaction; post that it is always initialized albeit with empty block - Fixes a bug in FTS delete document where we were incorrectly panicing - Fixes a bug in record segment apply materialization where for deletes and updates we missed writing the max offset id - Updates to metadata segment were missing updating the document and were only updating the metadata - Don't return error from the metadata segment if document is supplied as None for an update - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust None
*Summarize the changes made by this PR.* - Improvements & Bug fixes - Enables s3 retry by default - New functionality - None *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust None
ce5123c
to
3bcf445
Compare
let block = block_manager.get(&delta.id).await.unwrap(); | ||
// TODO: enable this assertion after the sizing is fixed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this TODO is not needed?
Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
pytest
for python,yarn test
for js,cargo test
for rustDocumentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?