Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smart filter block loading #3788

Open
RaduBerinde opened this issue Jul 25, 2024 · 0 comments
Open

smart filter block loading #3788

RaduBerinde opened this issue Jul 25, 2024 · 0 comments
Assignees
Labels
A-storage O-support P-3 Issues/test failures with no fix SLA T-storage

Comments

@RaduBerinde
Copy link
Member

RaduBerinde commented Jul 25, 2024

This is inspired from https:/cockroachlabs/support/issues/2992 and #3787

We can do a much better job at loading filter blocks. We currently either use them (blocking a Get) or don't use them at all. Ideally, we would load larger blocks that are valuable in the background.

Here's a proposal:

  • blocks not in L6 and under a certain size (e.g. 128KB) are loaded inline with the query, like today
  • larger blocks we do not load immediately but maintain a metadata-only cache for them, where we record how many times they would have been useful - i.e. how many SeekPrefixGE found no results with the prefix. Once we have enough "hits", we load the block in the background. The number of hits required should be proportional to the size (e.g. must have size/128KB recent "hits").
  • ideally for the block cache, only the times where the block actually helped exclude something should be counted as a "hit"

#3734 should also be kept in mind. We might eventually have multiple filter blocks of different sizes which we use in various combinations depending on level.

Jira issue: PEBBLE-226

@RaduBerinde RaduBerinde self-assigned this Jul 25, 2024
@nicktrav nicktrav added the P-3 Issues/test failures with no fix SLA label Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage O-support P-3 Issues/test failures with no fix SLA T-storage
Projects
Status: Backlog
Development

No branches or pull requests

2 participants