Add search_after to geonames and http_logs #130
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As we are optimizing "search_after" performance, we would like to measure it. This adds "search_after" operation to geonames and http_logs with a big enough value for "search_after". 1. geonames - "population" field. As most documents have 0 value for this field, search_after was added to asc sort. - "geonameid" field has unique values for each document. "search_after" was set to 5000000, as an approximate medium value. 2. http_logs - "@timestamp" field, "search_after" was set to "1998-06-10" for desc and asc sorts, which is a medium value.
|
Some results obtained on my laptop: http_logs Full results here geonames Full results here |
|
Also would like to ask @jimczi if using a median value for |
jimczi
requested changes
Sep 10, 2020
There was a problem hiding this comment.
I left some comments. Can you share the entire output for the challenges, it's truncated in your comments but I don't understand why the query is so costly. The fact that the throughput is too high shouldn't affect the service time too much.
Do you have an idea on how long the http_logs search_after query runs on a single node ?
| "sort" : [ | ||
| {"@timestamp" : "desc"} | ||
| ], | ||
| "search_after": ["1998-06-10"] |
| @@ -136,6 +136,20 @@ | |||
| "warmup-iterations": 200, | |||
| "iterations": 100, | |||
| "target-throughput": 2 | |||
| }, | |||
| { | |||
| "name": "desc-sort-with-after-timestamp-after-force-merge-1-seg", | |||
There was a problem hiding this comment.
Would be interesting to have the number before force merge too?
http_logs/challenges/default.json
Outdated
| { | ||
| "name": "desc-sort-with-after-timestamp-after-force-merge-1-seg", | ||
| "operation": "desc_sort_with_after_timestamp", | ||
| "warmup-iterations": 200, |
There was a problem hiding this comment.
these warmups and iterations are too big imo. 5 to 10 should be enough to have stable results ?
+1 to use a median value |
|
@jimczi Thank you for the feedback, at first I have incorrectly reported Also I've chatted with @jimczi offline, and we have defined the following tasks to be tackled:
|
|
@jimczi I have addressed your comments, and ran benchmarking once more with updated operations. Full results here |
|
@mayya-sharipova Thank you for the changes. I ran on our low-mem env and the target throughput looks okay to me. I will merge the request tomorrow and add the charts to https://elasticsearch-benchmarks.elastic.co later this week. Thanks! |
dliappis
pushed a commit
that referenced
this pull request
Nov 25, 2020
As we are optimizing "search_after" performance, we would like to measure it. This adds "search_after" operation to geonames and http_logs with a big enough value for "search_after". 1. geonames - "population" field. As most documents have 0 value for this field, search_after was added to asc sort. - "geonameid" field has unique values for each document. "search_after" was set to 5000000, as an approximate medium value. 2. http_logs - "@timestamp" field, "search_after" was set to "1998-06-10" for desc and asc sorts, which is a medium value.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As we are optimizing "search_after" performance, we would like to
measure it. This adds "search_after" operation to geonames
and http_logs with a big enough value for "search_after".
search_after was added to asc sort.
"search_after" was set to 5000000, as an approximate medium
value.
for desc and asc sorts, which is a medium value.