You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the Download call after an external ingestion (eg. online restore) schedules multiple copy compactions if viaBackingFileDownload is true, and rewrite compactions if not. It makes sense for the latter to still flow through the compaction flow as we're materializing any prefix/suffix rewrite rules as we're going. But for the former, we can and should offload the job of doing byte-by-byte copies of sstable blocks to the object provider, for two main reasons:
If we're doing block-by-block copies anyway, and data blocks are going to be identical between the remote and local sstable, we can avoid cache churn by keeping the file num and block handles identical between the remote and local file. This should reduce cache misses right after a download finishes.
The compaction picking process involves holding db.mu, a mutex that is also held by ingestions and rotations of the memtable. If we're doing high-throughput downloads, holding this mutex every time we pick one file to download is wasteful and can be avoided if we just do the download "below" the pebble Version.
Sub-issues filed under this meta issue will be listed below.
itsbilal
changed the title
db: offload downloads to objprovider to reduce cache and mutex churn
db: offload downloads to objprovider to reduce cache and db.mu churn
Jul 9, 2024
Currently, the
Download
call after an external ingestion (eg. online restore) schedules multiple copy compactions ifviaBackingFileDownload
is true, and rewrite compactions if not. It makes sense for the latter to still flow through the compaction flow as we're materializing any prefix/suffix rewrite rules as we're going. But for the former, we can and should offload the job of doing byte-by-byte copies of sstable blocks to the object provider, for two main reasons:db.mu
, a mutex that is also held by ingestions and rotations of the memtable. If we're doing high-throughput downloads, holding this mutex every time we pick one file to download is wasteful and can be avoided if we just do the download "below" the pebble Version.Sub-issues filed under this meta issue will be listed below.
Jira issue: PEBBLE-31
Epic CRDB-40359
The text was updated successfully, but these errors were encountered: