[Star tree] Performance optimizations during flush flow

[bharath-techie]

Description

1. Optimize metrics in off heap

Currently we initialize metric readers for all metric stats, thereby duplicating field value for each of the stats, when we write star tree document in off heap method.

If we initialize metric readers per metric field instead, we can write and read only the actual field values and when we read and initialize the star tree document, we can set the field's value to all metric stats.

2. Buffer complete star tree document instead of writing individual dimension / metrics to IndexOutput

We can buffer each starTreeDocument in memory and write bytes to indexOutput.

Related Issues

#16218

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 9476b03: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 86.31579% with 13 lines in your changes missing coverage. Please review.

Project coverage is 71.93%. Comparing base (74dba3f) to head (00bab32).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...acube/startree/builder/OffHeapStarTreeBuilder.java	60.00%	7 Missing and 1 partial ⚠️
...datacube/startree/builder/BaseStarTreeBuilder.java	86.36%	1 Missing and 2 partials ⚠️
...startree/builder/AbstractDocumentsFileManager.java	95.65%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16037      +/-   ##
============================================
+ Coverage     71.89%   71.93%   +0.03%     
- Complexity    64750    64758       +8     
============================================
  Files          5307     5307              
  Lines        302664   302714      +50     
  Branches      43726    43733       +7     
============================================
+ Hits         217612   217743     +131     
+ Misses        67183    67117      -66     
+ Partials      17869    17854      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for f77f98f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4e1a100: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4e1a100: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 00bab32: SUCCESS