You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe
Currently as part of star tree building during indexing, as part of off-heap build, we duplicate the metric field values for each metric stat, so if a metric field has 4 stats (sum, min, max, value_count) - we write 4 values into index output , instead of one.
Secondly we write individual dimension and metrics directly to IndexOutput.
Describe the solution you'd like
We can write only the actual field value from segments for each of the metric fields during flush.
We can buffer each starTreeDocument in memory and write bytes to indexOutput.
Star tree config
Dimensions
* Timestamp
* Minute
* Half Hour
* Target status code
* ELB status code
Metrics
* 8 field [ 4 stats each ]
Hers is how time taken to sort and aggregate segment documents to star-tree documents improved after these changes :
Baseline - 24 seconds for 1.5 million documents
Metric optimization - 10 seconds for 1.5 million documents
Star tree document buffer optimization + Metric optimization - 2.5 seconds for 1.5 million documents
Related component
Search:Performance
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe
Currently as part of star tree building during indexing, as part of off-heap build, we duplicate the metric field values for each metric stat, so if a metric field has 4 stats (sum, min, max, value_count) - we write 4 values into index output , instead of one.
Secondly we write individual dimension and metrics directly to IndexOutput.
Describe the solution you'd like
We can write only the actual field value from segments for each of the metric fields during flush.
We can buffer each starTreeDocument in memory and write bytes to indexOutput.
Hers is how time taken to sort and aggregate segment documents to star-tree documents improved after these changes :
Related component
Search:Performance
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: