Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add index on traceID #367

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

ankitnayan
Copy link
Contributor

adding index on traceID should enable searching by traceID much faster

@ankitnayan
Copy link
Contributor Author

this did not work in reducing granules scanned. Needs further investigation

@nityanandagohain
Copy link
Member

nityanandagohain commented Aug 21, 2024

try using bloom filter instead of tokenbf? also increase the size of the filter

@ankitnayan
Copy link
Contributor Author

try using bloom filter instead of tokenbf? also increase the size of the filter

Can you suggest the size to use?

@srikanthccv
Copy link
Member

@nityanandagohain please suggest the size. The #365 is waiting on this which will help in PRs like SigNoz/signoz#3184

@nityanandagohain
Copy link
Member

If a granule has around 4000 unique trace_id ( 8192 total rows),

The size of index = 10KB with 5 hash functions will give us probability of false positive 0.000314144 (1 in 3183).

https://hur.st/bloomfilter/?n=4000&p=&m=90000&k=5

let's start with 10KB and 5 hash functions ?

@ankitnayan
Copy link
Contributor Author

this PR can wait

Base automatically changed from chore/change-resourcemap-index to pull-364 September 8, 2024 14:17
Base automatically changed from pull-364 to main September 8, 2024 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants