perf(shared buffer): divide sync task to kr splits to sync shared buf…

[soundOfDestiny]

…fers parallel (close #1400)

What's changed and what's your intention?

divide sync task to kr splits to sync shared buffers parallel (close #1400)

Checklist

• I have written necessary docs and comments

Refer to a related PR or issue link (optional)

[codecov]

Codecov Report

Merging #1496 (bb6ec7d) into main (9f48392) will increase coverage by 0.17%.
The diff coverage is 72.97%.

@@             Coverage Diff              @@
##               main    #1496      +/-   ##
============================================
+ Coverage     69.70%   69.87%   +0.17%     
  Complexity     2766     2766              
============================================
  Files          1042     1042              
  Lines         91601    91813     +212     
  Branches       1790     1790              
============================================
+ Hits          63851    64158     +307     
+ Misses        26859    26764      -95     
  Partials        891      891              

Flag	Coverage Δ
java	61.01% <ø> (ø)
rust	71.73% <72.97%> (+0.20%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/bench/ss_bench/main.rs	0.95% <0.00%> (-0.01%)	⬇️
src/storage/src/hummock/compactor.rs	46.15% <69.84%> (+3.37%)	⬆️
src/common/src/config.rs	77.67% <100.00%> (+0.61%)	⬆️
...rc/hummock/shared_buffer/shared_buffer_uploader.rs	78.89% <100.00%> (-0.39%)	⬇️
src/storage/src/hummock/test_utils.rs	81.17% <100.00%> (+0.22%)	⬆️
src/storage/src/hummock/key_range.rs	58.53% <0.00%> (-8.54%)	⬇️
src/expr/src/vector_op/arithmetic_op.rs	77.96% <0.00%> (-5.06%)	⬇️
src/frontend/src/handler/mod.rs	41.93% <0.00%> (-2.90%)	⬇️
...rc/frontend/src/optimizer/property/distribution.rs	84.61% <0.00%> (-2.57%)	⬇️
src/frontend/src/binder/window_table_function.rs	68.60% <0.00%> (-2.05%)	⬇️
... and 56 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[twocode]

I think generating merge iterator on shared buffer batch should also be handled by build_sst_iter so that this part of code can be reused with compact() to initiate parallel compaction.
But this can be done in the next refactor.

[soundOfDestiny]

I think generating merge iterator on shared buffer batch should also be handled by build_sst_iter so that this part of code can be reused with compact() to initiate parallel compaction.
But this can be done in the next refactor.

seems hard. Giving facts that

• build_sst_iter is async
• Compactor needs to be cloned and then sent to different closures (and finally different threads)

thus we can't call build_sst_iter out of closure. But if we call build_sst_iter inside the closure, build_sst_iter in shared buffer situation needs a Vec<SharedBufferBatch>, which is not Arc

[twocode]

Looks good for the rest

[twocode]

LGTM for the rest. Cool!