Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Memory usage of hierarchical results normalizer is not accounted for #2244

Open
droberts195 opened this issue Mar 24, 2022 · 0 comments

Comments

@droberts195
Copy link
Contributor

We do not account for the memory size of the hierarchical results normalizer.

When it was first introduced it was fairly close to being a fixed size object, so could reasonably be considered part of the "process overhead" - fixed data structures for which we add 10MB to the required memory of each job.

However, now that we do per partition scoring it is no longer reasonable to consider the normalizer part of the fixed overhead. For a job with a large number of partition field values it can be a non-negligible size, and this memory should be properly accounted for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant