Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tuning and combined improvements for perf and observability #1493

Merged
merged 1 commit into from
May 14, 2024

Conversation

AhmedSoliman
Copy link
Contributor

@AhmedSoliman AhmedSoliman commented May 1, 2024

Copy link

github-actions bot commented May 1, 2024

Test Results

 99 files  ±0   99 suites  ±0   8m 23s ⏱️ -5s
 84 tests ±0   82 ✅ ±0  2 💤 ±0  0 ❌ ±0 
216 runs  ±0  210 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit cfe427d. ± Comparison against base commit a7e81b0.

♻️ This comment has been updated with latest results.

@AhmedSoliman AhmedSoliman changed the title Don't require multiple write buffers to merge Tuning and combined improvements for perf and observability May 2, 2024
@AhmedSoliman AhmedSoliman marked this pull request as ready for review May 2, 2024 17:42
Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for creating this PR @AhmedSoliman. The changes look good to me. I think the stall detector will be really useful :-) Also the additional performance metrics are super helpful.

My main question is how much the different RocksDB users are already respecting the calculated write buffer sizes in RocksDbOptions and whether there are planned follow-ups. Apart from this, +1 for merging.

crates/rocksdb/src/db_manager.rs Outdated Show resolved Hide resolved
crates/rocksdb/src/db_manager.rs Outdated Show resolved Hide resolved
crates/rocksdb/src/perf.rs Outdated Show resolved Hide resolved
crates/rocksdb/src/perf.rs Outdated Show resolved Hide resolved
crates/types/src/config/common.rs Show resolved Hide resolved
crates/types/src/config/rocksdb.rs Outdated Show resolved Hide resolved
Comment on lines +115 to +121
let common_opts = CommonOptions::default();
let all_memtables = common_opts.rocksdb_total_memtables_size();
let num_partitions = WorkerOptions::default().bootstrap_num_partitions();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess these numbers should not be taken from the defaults but from the actually configured values, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those values calculate the default value. I'll add a comment explaining that, but in a possible future, those will be calculated dynamically based on the actual system values (i.e. the current number of configured partitions). We cannot rely on bootstrap_num_partitions() because the system can diverge from this value after initial bootstrap.

Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :-) +1 for merging.

crates/types/src/config/rocksdb.rs Outdated Show resolved Hide resolved
@AhmedSoliman AhmedSoliman merged commit 4c829d0 into main May 14, 2024
10 checks passed
@AhmedSoliman AhmedSoliman deleted the pr1493 branch May 14, 2024 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants