diff --git a/docs/changelog/85649.yaml b/docs/changelog/85649.yaml new file mode 100644 index 0000000000000..8a3fcde09ab54 --- /dev/null +++ b/docs/changelog/85649.yaml @@ -0,0 +1,5 @@ +pr: 85649 +summary: Synthetic source +area: Mapping +type: feature +issues: [] diff --git a/docs/reference/search/profile.asciidoc b/docs/reference/search/profile.asciidoc index 22182b51ee53d..67918710d4728 100644 --- a/docs/reference/search/profile.asciidoc +++ b/docs/reference/search/profile.asciidoc @@ -6,22 +6,22 @@ WARNING: The Profile API is a debugging tool and adds significant overhead to search execution. -Provides detailed timing information about the execution of individual +Provides detailed timing information about the execution of individual components in a search request. [[search-profile-api-desc]] ==== {api-description-title} -The Profile API gives the user insight into how search requests are executed at -a low level so that the user can understand why certain requests are slow, and -take steps to improve them. Note that the Profile API, -<>, doesn't measure network latency, +The Profile API gives the user insight into how search requests are executed at +a low level so that the user can understand why certain requests are slow, and +take steps to improve them. Note that the Profile API, +<>, doesn't measure network latency, time spent while the requests spends in queues, or while merging shard responses on the coordinating node. -The output from the Profile API is *very* verbose, especially for complicated -requests executed across many shards. Pretty-printing the response is +The output from the Profile API is *very* verbose, especially for complicated +requests executed across many shards. Pretty-printing the response is recommended to help understand the output. @@ -172,7 +172,9 @@ The API returns the following result: "next_reader": 7292, "next_reader_count": 1, "load_stored_fields": 299325, - "load_stored_fields_count": 5 + "load_stored_fields_count": 5, + "load_source": 3863, + "load_source_count": 5 }, "debug": { "stored_fields": ["_id", "_routing", "_source"] @@ -206,7 +208,7 @@ The API returns the following result: <1> Search results are returned, but were omitted here for brevity. -Even for a simple query, the response is relatively complicated. Let's break it +Even for a simple query, the response is relatively complicated. Let's break it down piece-by-piece before moving to more complex examples. @@ -240,7 +242,7 @@ The overall structure of the profile response is as follows: // TESTRESPONSE[s/"collector": \[...\]/"collector": $body.$_path/] // TESTRESPONSE[s/"aggregations": \[...\]/"aggregations": []/] // TESTRESPONSE[s/"fetch": \{...\}/"fetch": $body.$_path/] -<1> A profile is returned for each shard that participated in the response, and +<1> A profile is returned for each shard that participated in the response, and is identified by a unique ID. <2> Query timings and other debugging information. <3> The cumulative rewrite time. @@ -248,26 +250,26 @@ is identified by a unique ID. <5> Aggregation timings, invocation counts, and debug information. <6> Fetch timing and debug information. -Because a search request may be executed against one or more shards in an index, -and a search may cover one or more indices, the top level element in the profile -response is an array of `shard` objects. Each shard object lists its `id` which -uniquely identifies the shard. The ID's format is +Because a search request may be executed against one or more shards in an index, +and a search may cover one or more indices, the top level element in the profile +response is an array of `shard` objects. Each shard object lists its `id` which +uniquely identifies the shard. The ID's format is `[nodeID][indexName][shardID]`. -The profile itself may consist of one or more "searches", where a search is a -query executed against the underlying Lucene index. Most search requests -submitted by the user will only execute a single `search` against the Lucene -index. But occasionally multiple searches will be executed, such as including a -global aggregation (which needs to execute a secondary "match_all" query for the +The profile itself may consist of one or more "searches", where a search is a +query executed against the underlying Lucene index. Most search requests +submitted by the user will only execute a single `search` against the Lucene +index. But occasionally multiple searches will be executed, such as including a +global aggregation (which needs to execute a secondary "match_all" query for the global context). Inside each `search` object there will be two arrays of profiled information: -a `query` array and a `collector` array. Alongside the `search` object is an -`aggregations` object that contains the profile information for the -aggregations. In the future, more sections may be added, such as `suggest`, +a `query` array and a `collector` array. Alongside the `search` object is an +`aggregations` object that contains the profile information for the +aggregations. In the future, more sections may be added, such as `suggest`, `highlight`, etc. -There will also be a `rewrite` metric showing the total time spent rewriting the +There will also be a `rewrite` metric showing the total time spent rewriting the query (in nanoseconds). NOTE: As with other statistics apis, the Profile API supports human readable outputs. This can be turned on by adding @@ -293,10 +295,10 @@ the `advance` phase of that query is the cause, for example. [[query-section]] ===== `query` Section -The `query` section contains detailed timing of the query tree executed by -Lucene on a particular shard. The overall structure of this query tree will -resemble your original Elasticsearch query, but may be slightly (or sometimes -very) different. It will also use similar but not always identical naming. +The `query` section contains detailed timing of the query tree executed by +Lucene on a particular shard. The overall structure of this query tree will +resemble your original Elasticsearch query, but may be slightly (or sometimes +very) different. It will also use similar but not always identical naming. Using our previous `match` query example, let's analyze the `query` section: [source,console-result] @@ -330,27 +332,27 @@ Using our previous `match` query example, let's analyze the `query` section: // TESTRESPONSE[s/"breakdown": \{...\}/"breakdown": $body.$_path/] <1> The breakdown timings are omitted for simplicity. -Based on the profile structure, we can see that our `match` query was rewritten -by Lucene into a BooleanQuery with two clauses (both holding a TermQuery). The -`type` field displays the Lucene class name, and often aligns with the -equivalent name in Elasticsearch. The `description` field displays the Lucene -explanation text for the query, and is made available to help differentiating -between parts of your query (e.g. both `message:get` and `message:search` are +Based on the profile structure, we can see that our `match` query was rewritten +by Lucene into a BooleanQuery with two clauses (both holding a TermQuery). The +`type` field displays the Lucene class name, and often aligns with the +equivalent name in Elasticsearch. The `description` field displays the Lucene +explanation text for the query, and is made available to help differentiating +between parts of your query (e.g. both `message:get` and `message:search` are TermQuery's and would appear identical otherwise. -The `time_in_nanos` field shows that this query took ~11.9ms for the entire +The `time_in_nanos` field shows that this query took ~11.9ms for the entire BooleanQuery to execute. The recorded time is inclusive of all children. -The `breakdown` field will give detailed stats about how the time was spent, -we'll look at that in a moment. Finally, the `children` array lists any -sub-queries that may be present. Because we searched for two values ("get -search"), our BooleanQuery holds two children TermQueries. They have identical -information (type, time, breakdown, etc). Children are allowed to have their +The `breakdown` field will give detailed stats about how the time was spent, +we'll look at that in a moment. Finally, the `children` array lists any +sub-queries that may be present. Because we searched for two values ("get +search"), our BooleanQuery holds two children TermQueries. They have identical +information (type, time, breakdown, etc). Children are allowed to have their own children. ===== Timing Breakdown -The `breakdown` component lists detailed timing statistics about low-level +The `breakdown` component lists detailed timing statistics about low-level Lucene execution: [source,console-result] @@ -380,11 +382,11 @@ Lucene execution: // TESTRESPONSE[s/}$/},\n"children": $body.$_path}],\n"rewrite_time": $body.$_path, "collector": $body.$_path}], "aggregations": [], "fetch": $body.$_path}]}}/] // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/] -Timings are listed in wall-clock nanoseconds and are not normalized at all. All -caveats about the overall `time_in_nanos` apply here. The intention of the -breakdown is to give you a feel for A) what machinery in Lucene is actually -eating time, and B) the magnitude of differences in times between the various -components. Like the overall time, the breakdown is inclusive of all children +Timings are listed in wall-clock nanoseconds and are not normalized at all. All +caveats about the overall `time_in_nanos` apply here. The intention of the +breakdown is to give you a feel for A) what machinery in Lucene is actually +eating time, and B) the magnitude of differences in times between the various +components. Like the overall time, the breakdown is inclusive of all children times. The meaning of the stats are as follows: @@ -459,10 +461,10 @@ The meaning of the stats are as follows: [[collectors-section]] ===== `collectors` Section -The Collectors portion of the response shows high-level execution details. -Lucene works by defining a "Collector" which is responsible for coordinating the -traversal, scoring, and collection of matching documents. Collectors are also -how a single query can record aggregation results, execute unscoped "global" +The Collectors portion of the response shows high-level execution details. +Lucene works by defining a "Collector" which is responsible for coordinating the +traversal, scoring, and collection of matching documents. Collectors are also +how a single query can record aggregation results, execute unscoped "global" queries, execute post-query filters, etc. Looking at the previous example: @@ -482,18 +484,18 @@ Looking at the previous example: // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/] -We see a single collector named `SimpleTopScoreDocCollector` wrapped into -`CancellableCollector`. `SimpleTopScoreDocCollector` is the default "scoring and -sorting" `Collector` used by {es}. The `reason` field attempts to give a plain -English description of the class name. The `time_in_nanos` is similar to the -time in the Query tree: a wall-clock time inclusive of all children. Similarly, -`children` lists all sub-collectors. The `CancellableCollector` that wraps -`SimpleTopScoreDocCollector` is used by {es} to detect if the current search was +We see a single collector named `SimpleTopScoreDocCollector` wrapped into +`CancellableCollector`. `SimpleTopScoreDocCollector` is the default "scoring and +sorting" `Collector` used by {es}. The `reason` field attempts to give a plain +English description of the class name. The `time_in_nanos` is similar to the +time in the Query tree: a wall-clock time inclusive of all children. Similarly, +`children` lists all sub-collectors. The `CancellableCollector` that wraps +`SimpleTopScoreDocCollector` is used by {es} to detect if the current search was cancelled and stop collecting documents as soon as it occurs. -It should be noted that Collector times are **independent** from the Query -times. They are calculated, combined, and normalized independently! Due to the -nature of Lucene's execution, it is impossible to "merge" the times from the +It should be noted that Collector times are **independent** from the Query +times. They are calculated, combined, and normalized independently! Due to the +nature of Lucene's execution, it is impossible to "merge" the times from the Collectors into the Query section, so they are displayed in separate portions. For reference, the various collector reasons are: @@ -545,21 +547,21 @@ For reference, the various collector reasons are: [[rewrite-section]] ===== `rewrite` Section -All queries in Lucene undergo a "rewriting" process. A query (and its -sub-queries) may be rewritten one or more times, and the process continues until -the query stops changing. This process allows Lucene to perform optimizations, -such as removing redundant clauses, replacing one query for a more efficient -execution path, etc. For example a Boolean -> Boolean -> TermQuery can be +All queries in Lucene undergo a "rewriting" process. A query (and its +sub-queries) may be rewritten one or more times, and the process continues until +the query stops changing. This process allows Lucene to perform optimizations, +such as removing redundant clauses, replacing one query for a more efficient +execution path, etc. For example a Boolean -> Boolean -> TermQuery can be rewritten to a TermQuery, because all the Booleans are unnecessary in this case. -The rewriting process is complex and difficult to display, since queries can -change drastically. Rather than showing the intermediate results, the total -rewrite time is simply displayed as a value (in nanoseconds). This value is +The rewriting process is complex and difficult to display, since queries can +change drastically. Rather than showing the intermediate results, the total +rewrite time is simply displayed as a value (in nanoseconds). This value is cumulative and contains the total time for all queries being rewritten. ===== A more complex example -To demonstrate a slightly more complex query and the associated results, we can +To demonstrate a slightly more complex query and the associated results, we can profile the following query: [source,console] @@ -715,44 +717,44 @@ The API returns the following result: // TESTRESPONSE[s/\.\.\.//] // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/] // TESTRESPONSE[s/"id": "\[P6-vulHtQRWuD4YnubWb7A\]\[my-index-000001\]\[0\]"/"id": $body.profile.shards.0.id/] -<1> The `"aggregations"` portion has been omitted because it will be covered in +<1> The `"aggregations"` portion has been omitted because it will be covered in the next section. -As you can see, the output is significantly more verbose than before. All the +As you can see, the output is significantly more verbose than before. All the major portions of the query are represented: 1. The first `TermQuery` (user.id:elkbee) represents the main `term` query. 2. The second `TermQuery` (message:search) represents the `post_filter` query. -The Collector tree is fairly straightforward, showing how a single -CancellableCollector wraps a MultiCollector which also wraps a FilteredCollector -to execute the post_filter (and in turn wraps the normal scoring +The Collector tree is fairly straightforward, showing how a single +CancellableCollector wraps a MultiCollector which also wraps a FilteredCollector +to execute the post_filter (and in turn wraps the normal scoring SimpleCollector), a BucketCollector to run all scoped aggregations. ===== Understanding MultiTermQuery output -A special note needs to be made about the `MultiTermQuery` class of queries. -This includes wildcards, regex, and fuzzy queries. These queries emit very +A special note needs to be made about the `MultiTermQuery` class of queries. +This includes wildcards, regex, and fuzzy queries. These queries emit very verbose responses, and are not overly structured. -Essentially, these queries rewrite themselves on a per-segment basis. If you -imagine the wildcard query `b*`, it technically can match any token that begins -with the letter "b". It would be impossible to enumerate all possible -combinations, so Lucene rewrites the query in context of the segment being -evaluated, e.g., one segment may contain the tokens `[bar, baz]`, so the query -rewrites to a BooleanQuery combination of "bar" and "baz". Another segment may -only have the token `[bakery]`, so the query rewrites to a single TermQuery for +Essentially, these queries rewrite themselves on a per-segment basis. If you +imagine the wildcard query `b*`, it technically can match any token that begins +with the letter "b". It would be impossible to enumerate all possible +combinations, so Lucene rewrites the query in context of the segment being +evaluated, e.g., one segment may contain the tokens `[bar, baz]`, so the query +rewrites to a BooleanQuery combination of "bar" and "baz". Another segment may +only have the token `[bakery]`, so the query rewrites to a single TermQuery for "bakery". -Due to this dynamic, per-segment rewriting, the clean tree structure becomes -distorted and no longer follows a clean "lineage" showing how one query rewrites -into the next. At present time, all we can do is apologize, and suggest you -collapse the details for that query's children if it is too confusing. Luckily, -all the timing statistics are correct, just not the physical layout in the +Due to this dynamic, per-segment rewriting, the clean tree structure becomes +distorted and no longer follows a clean "lineage" showing how one query rewrites +into the next. At present time, all we can do is apologize, and suggest you +collapse the details for that query's children if it is too confusing. Luckily, +all the timing statistics are correct, just not the physical layout in the response, so it is sufficient to just analyze the top-level MultiTermQuery and ignore its children if you find the details too tricky to interpret. -Hopefully this will be fixed in future iterations, but it is a tricky problem to +Hopefully this will be fixed in future iterations, but it is a tricky problem to solve and still in-progress. :) [[profiling-aggregations]] @@ -763,9 +765,9 @@ solve and still in-progress. :) ====== `aggregations` Section -The `aggregations` section contains detailed timing of the aggregation tree -executed by a particular shard. The overall structure of this aggregation tree -will resemble your original {es} request. Let's execute the previous query again +The `aggregations` section contains detailed timing of the aggregation tree +executed by a particular shard. The overall structure of this aggregation tree +will resemble your original {es} request. Let's execute the previous query again and look at the aggregation profile this time: [source,console] @@ -899,14 +901,14 @@ This yields the following aggregation profile output: // TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/] // TESTRESPONSE[s/"id": "\[P6-vulHtQRWuD4YnubWb7A\]\[my-index-000001\]\[0\]"/"id": $body.profile.shards.0.id/] -From the profile structure we can see that the `my_scoped_agg` is internally -being run as a `NumericTermsAggregator` (because the field it is aggregating, -`http.response.status_code`, is a numeric field). At the same level, we see a `GlobalAggregator` -which comes from `my_global_agg`. That aggregation then has a child +From the profile structure we can see that the `my_scoped_agg` is internally +being run as a `NumericTermsAggregator` (because the field it is aggregating, +`http.response.status_code`, is a numeric field). At the same level, we see a `GlobalAggregator` +which comes from `my_global_agg`. That aggregation then has a child `NumericTermsAggregator` which comes from the second term's aggregation on `http.response.status_code`. -The `time_in_nanos` field shows the time executed by each aggregation, and is -inclusive of all children. While the overall time is useful, the `breakdown` +The `time_in_nanos` field shows the time executed by each aggregation, and is +inclusive of all children. While the overall time is useful, the `breakdown` field will give detailed stats about how the time was spent. Some aggregations may return expert `debug` information that describe features @@ -944,10 +946,10 @@ method. For example, `"collect_count": 2` means the aggregation called the `collect()` on two different documents. The `reduce` property is reserved for future use and always returns `0`. -Timings are listed in wall-clock nanoseconds and are not normalized at all. All -caveats about the overall `time` apply here. The intention of the breakdown is -to give you a feel for A) what machinery in {es} is actually eating time, and B) -the magnitude of differences in times between the various components. Like the +Timings are listed in wall-clock nanoseconds and are not normalized at all. All +caveats about the overall `time` apply here. The intention of the breakdown is +to give you a feel for A) what machinery in {es} is actually eating time, and B) +the magnitude of differences in times between the various components. Like the overall time, the breakdown is inclusive of all children times. [[profiling-fetch]] @@ -989,7 +991,9 @@ And here is the fetch profile: "next_reader": 7292, "next_reader_count": 1, "load_stored_fields": 299325, - "load_stored_fields_count": 5 + "load_stored_fields_count": 5, + "load_source": 3863, + "load_source_count": 5 }, "debug": { "stored_fields": ["_id", "_routing", "_source"] @@ -1046,24 +1050,24 @@ loading stored fields by setting [[profiling-considerations]] ===== Profiling Considerations -Like any profiler, the Profile API introduces a non-negligible overhead to -search execution. The act of instrumenting low-level method calls such as -`collect`, `advance`, and `next_doc` can be fairly expensive, since these -methods are called in tight loops. Therefore, profiling should not be enabled -in production settings by default, and should not be compared against +Like any profiler, the Profile API introduces a non-negligible overhead to +search execution. The act of instrumenting low-level method calls such as +`collect`, `advance`, and `next_doc` can be fairly expensive, since these +methods are called in tight loops. Therefore, profiling should not be enabled +in production settings by default, and should not be compared against non-profiled query times. Profiling is just a diagnostic tool. -There are also cases where special Lucene optimizations are disabled, since they -are not amenable to profiling. This could cause some queries to report larger -relative times than their non-profiled counterparts, but in general should not +There are also cases where special Lucene optimizations are disabled, since they +are not amenable to profiling. This could cause some queries to report larger +relative times than their non-profiled counterparts, but in general should not have a drastic effect compared to other components in the profiled query. [[profile-limitations]] ===== Limitations - Profiling currently does not measure the network overhead. -- Profiling also does not account for time spent in the queue, merging shard -responses on the coordinating node, or additional work such as building global +- Profiling also does not account for time spent in the queue, merging shard +responses on the coordinating node, or additional work such as building global ordinals (an internal data structure used to speed up search). - Profiling statistics are currently not available for suggestions, highlighting, `dfs_query_then_fetch`. diff --git a/modules/legacy-geo/src/test/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapperTests.java b/modules/legacy-geo/src/test/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapperTests.java index 15899368f0fe5..5f4c8f0afd3e8 100644 --- a/modules/legacy-geo/src/test/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapperTests.java +++ b/modules/legacy-geo/src/test/java/org/elasticsearch/legacygeo/mapper/LegacyGeoShapeFieldMapperTests.java @@ -34,6 +34,7 @@ import org.elasticsearch.test.VersionUtils; import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -677,4 +678,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java index 19c99fbc64427..c1fce4824f372 100644 --- a/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java +++ b/modules/mapper-extras/src/main/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapper.java @@ -33,6 +33,7 @@ import org.elasticsearch.index.mapper.MapperBuilderContext; import org.elasticsearch.index.mapper.NumberFieldMapper; import org.elasticsearch.index.mapper.SimpleMappedFieldType; +import org.elasticsearch.index.mapper.SourceLoader; import org.elasticsearch.index.mapper.SourceValueFetcher; import org.elasticsearch.index.mapper.TextSearchInfo; import org.elasticsearch.index.mapper.TimeSeriesParams; @@ -349,6 +350,20 @@ private double scale(Object input) { public TimeSeriesParams.MetricType getMetricType() { return metricType; } + + @Override + public String toString() { + StringBuilder b = new StringBuilder(); + b.append("ScaledFloatFieldType[").append(scalingFactor); + if (nullValue != null) { + b.append(", nullValue=").append(nullValue); + ; + } + if (metricType != null) { + b.append(", metricType=").append(metricType); + } + return b.append("]").toString(); + } } private final Explicit ignoreMalformed; @@ -641,4 +656,29 @@ public Object nextValue() throws IOException { }; } } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (hasDocValues == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (ignoreMalformed.value()) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it ignores malformed numbers" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + return new NumberFieldMapper.NumericSyntheticFieldLoader(name(), simpleName()) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(value / scalingFactor); + } + }; + } } diff --git a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapperTests.java b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapperTests.java index faf89871044c3..6a7270fe8f5bc 100644 --- a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapperTests.java +++ b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/MatchOnlyTextFieldMapperTests.java @@ -28,6 +28,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.hamcrest.Matchers; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -162,4 +163,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { protected void randomFetchTestFieldConfig(XContentBuilder b) throws IOException { assumeFalse("We don't have a way to assert things here", true); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapperTests.java b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapperTests.java index 39e5c915bbd6a..b906803b04c8c 100644 --- a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapperTests.java +++ b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeatureFieldMapperTests.java @@ -23,6 +23,7 @@ import org.elasticsearch.index.mapper.ParsedDocument; import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -157,4 +158,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapperTests.java b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapperTests.java index bb3e3141497e3..d360b19e7d0a3 100644 --- a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapperTests.java +++ b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/RankFeaturesFieldMapperTests.java @@ -20,6 +20,7 @@ import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; import org.hamcrest.Matchers; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -176,4 +177,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { protected boolean allowsNullValues() { return false; // TODO should this allow null values? } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapperTests.java b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapperTests.java index 3435fefa8c9de..79b152d986b9e 100644 --- a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapperTests.java +++ b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/ScaledFloatFieldMapperTests.java @@ -8,10 +8,12 @@ package org.elasticsearch.index.mapper.extras; +import org.apache.lucene.index.DirectoryReader; import org.apache.lucene.index.DocValuesType; import org.apache.lucene.index.IndexableField; import org.elasticsearch.common.Strings; import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.core.Tuple; import org.elasticsearch.index.mapper.DocumentMapper; import org.elasticsearch.index.mapper.MappedFieldType; import org.elasticsearch.index.mapper.MapperParsingException; @@ -23,6 +25,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentType; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -31,6 +34,7 @@ import static java.util.Collections.singletonList; import static org.hamcrest.Matchers.containsString; +import static org.hamcrest.Matchers.equalTo; public class ScaledFloatFieldMapperTests extends MapperTestCase { @@ -349,4 +353,83 @@ protected Object generateRandomInputValue(MappedFieldType ft) { default -> throw new IllegalArgumentException(); }; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new SyntheticSourceSupport() { + private final double scalingFactor = randomDoubleBetween(0, Double.MAX_VALUE, false); + private final Double nullValue = usually() ? null : round(randomValue()); + + @Override + public SyntheticSourceExample example() { + if (randomBoolean()) { + Tuple v = generateValue(); + return new SyntheticSourceExample(v.v1(), v.v2(), this::mapping); + } + List> values = randomList(1, 5, this::generateValue); + List in = values.stream().map(Tuple::v1).toList(); + List outList = values.stream().map(Tuple::v2).sorted().toList(); + Object out = outList.size() == 1 ? outList.get(0) : outList; + return new SyntheticSourceExample(in, out, this::mapping); + } + + private Tuple generateValue() { + if (nullValue != null && randomBoolean()) { + return Tuple.tuple(null, nullValue); + } + double d = randomValue(); + return Tuple.tuple(d, round(d)); + } + + private double randomValue() { + return randomBoolean() ? randomDoubleBetween(-Double.MAX_VALUE, Double.MAX_VALUE, true) : randomFloat(); + } + + private double round(double d) { + long encoded = Math.round(d * scalingFactor); + return encoded / scalingFactor; + } + + private void mapping(XContentBuilder b) throws IOException { + b.field("type", "scaled_float"); + b.field("scaling_factor", scalingFactor); + if (nullValue != null) { + b.field("null_value", nullValue); + } + if (rarely()) { + b.field("index", false); + } + if (rarely()) { + b.field("store", false); + } + } + + @Override + public List invalidExample() throws IOException { + return List.of( + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [scaled_float] doesn't support synthetic source because it doesn't have doc values"), + b -> b.field("type", "scaled_float").field("scaling_factor", 10).field("doc_values", false) + ), + new SyntheticSourceInvalidExample( + equalTo( + "field [field] of type [scaled_float] doesn't support synthetic source because it ignores malformed numbers" + ), + b -> b.field("type", "scaled_float").field("scaling_factor", 10).field("ignore_malformed", true) + ) + ); + } + }; + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected void validateRoundTripReader(String syntheticSource, DirectoryReader reader, DirectoryReader roundTripReader) + throws IOException { + // Intentionally disabled because it doesn't work yet + } } diff --git a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapperTests.java b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapperTests.java index 1ec3a7dfb7dd5..93bdba3273167 100644 --- a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapperTests.java +++ b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/SearchAsYouTypeFieldMapperTests.java @@ -54,6 +54,7 @@ import org.elasticsearch.index.search.QueryStringQueryParser; import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.ArrayList; @@ -793,4 +794,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("We don't have doc values or fielddata", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapperTests.java b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapperTests.java index 7d1aa87b1341d..c706410c17156 100644 --- a/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapperTests.java +++ b/modules/mapper-extras/src/test/java/org/elasticsearch/index/mapper/extras/TokenCountFieldMapperTests.java @@ -26,6 +26,7 @@ import org.elasticsearch.index.mapper.SourceToParse; import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -186,4 +187,14 @@ protected String generateRandomInputValue(MappedFieldType ft) { protected void randomFetchTestFieldConfig(XContentBuilder b) throws IOException { b.field("type", "token_count").field("analyzer", "standard"); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/modules/parent-join/src/yamlRestTest/resources/rest-api-spec/test/60_synthetic_source.yml b/modules/parent-join/src/yamlRestTest/resources/rest-api-spec/test/60_synthetic_source.yml new file mode 100644 index 0000000000000..5a86fa2074675 --- /dev/null +++ b/modules/parent-join/src/yamlRestTest/resources/rest-api-spec/test/60_synthetic_source.yml @@ -0,0 +1,18 @@ +unsupported: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + catch: bad_request + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + join_field: + type: join + relations: + parent: child diff --git a/modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/reindex/110_synthetic_source.yml b/modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/reindex/110_synthetic_source.yml new file mode 100644 index 0000000000000..8c1cf9eb328ce --- /dev/null +++ b/modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/reindex/110_synthetic_source.yml @@ -0,0 +1,146 @@ +setup: + - do: + indices.create: + index: synthetic + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + + - do: + indices.create: + index: standard + body: + mappings: + properties: + kwd: + type: keyword + +--- +from synthetic: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + bulk: + refresh: true + index: synthetic + body: + - '{"index": {}}' + - '{"kwd": "aaa"}' + - '{"index": {}}' + - '{"kwd": "bbb"}' + - '{"index": {}}' + - '{"kwd": "ccc"}' + - '{"index": {}}' + - '{"kwd": "ddd"}' + - '{"index": {}}' + - '{"kwd": "eee"}' + + - do: + reindex: + refresh: true + body: + source: + index: synthetic + dest: + index: standard + - match: {created: 5} + - match: {updated: 0} + - match: {version_conflicts: 0} + - match: {batches: 1} + - match: {failures: []} + - match: {throttled_millis: 0} + - gte: { took: 0 } + - is_false: task + - is_false: deleted + + - do: + search: + index: standard + body: + sort: + - kwd: asc + - match: { hits.total.value: 5 } + - match: + hits.hits.0._source: + kwd: aaa + - match: + hits.hits.1._source: + kwd: bbb + - match: + hits.hits.2._source: + kwd: ccc + - match: + hits.hits.3._source: + kwd: ddd + - match: + hits.hits.4._source: + kwd: eee + +--- +from standard: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + bulk: + refresh: true + index: standard + body: + - '{"index": {}}' + - '{"kwd": "aaa"}' + - '{"index": {}}' + - '{"kwd": "bbb"}' + - '{"index": {}}' + - '{"kwd": "ccc"}' + - '{"index": {}}' + - '{"kwd": "ddd"}' + - '{"index": {}}' + - '{"kwd": "eee"}' + + - do: + reindex: + refresh: true + body: + source: + index: standard + dest: + index: synthetic + - match: {created: 5} + - match: {updated: 0} + - match: {version_conflicts: 0} + - match: {batches: 1} + - match: {failures: []} + - match: {throttled_millis: 0} + - gte: { took: 0 } + - is_false: task + - is_false: deleted + + - do: + search: + index: synthetic + body: + sort: + - kwd: asc + - match: { hits.total.value: 5 } + - match: + hits.hits.0._source: + kwd: aaa + - match: + hits.hits.1._source: + kwd: bbb + - match: + hits.hits.2._source: + kwd: ccc + - match: + hits.hits.3._source: + kwd: ddd + - match: + hits.hits.4._source: + kwd: eee diff --git a/modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/update_by_query/100_synthetic_source.yml b/modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/update_by_query/100_synthetic_source.yml new file mode 100644 index 0000000000000..36c217297bd2f --- /dev/null +++ b/modules/reindex/src/yamlRestTest/resources/rest-api-spec/test/update_by_query/100_synthetic_source.yml @@ -0,0 +1,58 @@ +update: + - do: + indices.create: + index: synthetic + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + + - do: + bulk: + refresh: true + index: synthetic + body: + - '{"index": {}}' + - '{"kwd": "aaa", "int": 1}' + - '{"index": {}}' + - '{"kwd": "bbb", "int": 2}' + - '{"index": {}}' + - '{"kwd": "ccc", "int": 3}' + - '{"index": {}}' + - '{"kwd": "ccc", "int": 4}' + - '{"index": {}}' + - '{"kwd": "aaa", "int": 5}' + + - do: + update_by_query: + index: synthetic + refresh: true + body: + script: + lang: painless + source: ctx._source.int += 1 + + - do: + search: + index: synthetic + body: + size: 0 + aggs: + kwd: + terms: + field: kwd + aggs: + sum: + sum: + field: int + - match: {hits.total.value: 5} + - length: {aggregations.kwd.buckets: 3} + - match: {aggregations.kwd.buckets.0.key: aaa} + - match: {aggregations.kwd.buckets.0.sum.value: 8} + - match: {aggregations.kwd.buckets.1.key: ccc} + - match: {aggregations.kwd.buckets.1.sum.value: 9} + - match: {aggregations.kwd.buckets.2.key: bbb} + - match: {aggregations.kwd.buckets.2.sum.value: 3} diff --git a/plugins/analysis-icu/src/test/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapperTests.java b/plugins/analysis-icu/src/test/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapperTests.java index 232d182df3517..ebb69af101307 100644 --- a/plugins/analysis-icu/src/test/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapperTests.java +++ b/plugins/analysis-icu/src/test/java/org/elasticsearch/plugin/analysis/icu/ICUCollationKeywordFieldMapperTests.java @@ -29,6 +29,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentType; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -300,4 +301,14 @@ protected String generateRandomInputValue(MappedFieldType ft) { */ return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/plugins/mapper-annotated-text/src/internalClusterTest/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapperTests.java b/plugins/mapper-annotated-text/src/internalClusterTest/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapperTests.java index c0afbd5499a7d..e80a62f0dd512 100644 --- a/plugins/mapper-annotated-text/src/internalClusterTest/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapperTests.java +++ b/plugins/mapper-annotated-text/src/internalClusterTest/java/org/elasticsearch/index/mapper/annotatedtext/AnnotatedTextFieldMapperTests.java @@ -43,6 +43,7 @@ import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -587,4 +588,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("annotated_text doesn't have fielddata so we can't check against anything here.", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/plugins/mapper-murmur3/src/test/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapperTests.java b/plugins/mapper-murmur3/src/test/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapperTests.java index b02ec454d529a..36a93da92b365 100644 --- a/plugins/mapper-murmur3/src/test/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapperTests.java +++ b/plugins/mapper-murmur3/src/test/java/org/elasticsearch/index/mapper/murmur3/Murmur3FieldMapperTests.java @@ -18,6 +18,7 @@ import org.elasticsearch.plugin.mapper.MapperMurmur3Plugin; import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -62,4 +63,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/IndexingIT.java b/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/IndexingIT.java index 3e6bc10b89b1b..1e878495d5028 100644 --- a/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/IndexingIT.java +++ b/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/IndexingIT.java @@ -360,6 +360,76 @@ private void assertTsdbAgg(Matcher... expected) throws IOException { ); } + public void testSyntheticSource() throws IOException { + assumeTrue("added in 8.3.0", UPGRADE_FROM_VERSION.onOrAfter(Version.V_8_3_0)); + + switch (CLUSTER_TYPE) { + case OLD -> { + Request createIndex = new Request("PUT", "/synthetic"); + XContentBuilder indexSpec = XContentBuilder.builder(XContentType.JSON.xContent()).startObject(); + indexSpec.startObject("mappings"); + { + indexSpec.startObject("_source").field("synthetic", true).endObject(); + indexSpec.startObject("properties").startObject("kwd").field("type", "keyword").endObject().endObject(); + } + indexSpec.endObject(); + createIndex.setJsonEntity(Strings.toString(indexSpec.endObject())); + client().performRequest(createIndex); + bulk("synthetic", """ + {"index": {"_index": "synthetic", "_id": "old"}} + {"kwd": "old", "int": -12} + """); + break; + } + case MIXED -> { + if (FIRST_MIXED_ROUND) { + bulk("synthetic", """ + {"index": {"_index": "synthetic", "_id": "mixed_1"}} + {"kwd": "mixed_1", "int": 22} + """); + } else { + bulk("synthetic", """ + {"index": {"_index": "synthetic", "_id": "mixed_2"}} + {"kwd": "mixed_2", "int": 33} + """); + } + break; + } + case UPGRADED -> { + bulk("synthetic", """ + {"index": {"_index": "synthetic", "_id": "new"}} + {"kwd": "new", "int": 21341325} + """); + } + } + + assertMap( + entityAsMap(client().performRequest(new Request("GET", "/synthetic/_doc/old"))), + matchesMap().extraOk().entry("_source", matchesMap().entry("kwd", "old").entry("int", -12)) + ); + if (CLUSTER_TYPE == ClusterType.OLD) { + return; + } + assertMap( + entityAsMap(client().performRequest(new Request("GET", "/synthetic/_doc/mixed_1"))), + matchesMap().extraOk().entry("_source", matchesMap().entry("kwd", "mixed_1").entry("int", 22)) + ); + if (CLUSTER_TYPE == ClusterType.MIXED && FIRST_MIXED_ROUND) { + return; + } + assertMap( + entityAsMap(client().performRequest(new Request("GET", "/synthetic/_doc/mixed_2"))), + matchesMap().extraOk().entry("_source", matchesMap().entry("kwd", "mixed_2").entry("int", 33)) + ); + if (CLUSTER_TYPE == ClusterType.MIXED) { + return; + } + assertMap( + entityAsMap(client().performRequest(new Request("GET", "/synthetic/_doc/new"))), + matchesMap().extraOk().entry("_source", matchesMap().entry("kwd", "new").entry("int", 21341325)) + ); + } + private void assertCount(String index, int count) throws IOException { Request searchTestIndexRequest = new Request("POST", "/" + index + "/_search"); searchTestIndexRequest.addParameter(TOTAL_HITS_AS_INT_PARAM, "true"); diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/get/100_synthetic_source.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/get/100_synthetic_source.yml new file mode 100644 index 0000000000000..4c10574714980 --- /dev/null +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/get/100_synthetic_source.yml @@ -0,0 +1,35 @@ +keyword: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + + - do: + index: + index: test + id: 1 + refresh: true + body: + kwd: foo + + - do: + get: + index: test + id: 1 + - match: {_index: "test"} + - match: {_id: "1"} + - match: {_version: 1} + - match: {found: true} + - match: + _source: + kwd: foo diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/20_synthetic_source.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/20_synthetic_source.yml new file mode 100644 index 0000000000000..1a557cfd0c859 --- /dev/null +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/indices.create/20_synthetic_source.yml @@ -0,0 +1,38 @@ +invalid: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + catch: bad_request + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + doc_values: false + +--- +nested is disabled: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + catch: bad_request + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + n: + type: nested + properties: + foo: + type: keyword diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/mget/90_synthetic_source.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/mget/90_synthetic_source.yml new file mode 100644 index 0000000000000..05d7f41445601 --- /dev/null +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/mget/90_synthetic_source.yml @@ -0,0 +1,47 @@ +keyword: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + + - do: + index: + index: test + id: 1 + body: + kwd: foo + + - do: + index: + index: test + id: 2 + body: + kwd: bar + + + - do: + mget: + index: test + body: + ids: [1, 2] + - match: {docs.0._index: "test"} + - match: {docs.0._id: "1"} + - match: + docs.0._source: + kwd: foo + + - match: {docs.1._index: "test"} + - match: {docs.1._id: "2"} + - match: + docs.1._source: + kwd: bar diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/400_synthetic_source.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/400_synthetic_source.yml new file mode 100644 index 0000000000000..51d75e6980ca0 --- /dev/null +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/search/400_synthetic_source.yml @@ -0,0 +1,34 @@ +keyword: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + + - do: + index: + index: test + id: 1 + refresh: true + body: + kwd: foo + + - do: + search: + index: test + body: + query: + ids: + values: [1] + - match: + hits.hits.0._source: + kwd: foo diff --git a/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/update/100_synthetic_source.yml b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/update/100_synthetic_source.yml new file mode 100644 index 0000000000000..c0fb3096380d9 --- /dev/null +++ b/rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/update/100_synthetic_source.yml @@ -0,0 +1,56 @@ +keyword: + - skip: + version: " - 8.2.99" + reason: introduced in 8.3.0 + + - do: + indices.create: + index: test + body: + mappings: + _source: + synthetic: true + properties: + kwd: + type: keyword + kwd2: + type: keyword + + - do: + index: + index: test + id: 1 + refresh: true + body: + kwd: foo + + - do: + update: + index: test + id: 1 + body: + doc_as_upsert: true + doc: + kwd2: bar + - match: {result: updated} + + - do: + get: + index: test + id: 1 + - match: {_index: "test"} + - match: {_id: "1"} + - match: {_version: 2} + - match: {found: true} + - match: + _source: + kwd: foo + kwd2: bar + + # Make sure there isn't any _source stored field + - do: + indices.disk_usage: + index: test + run_expensive_tasks: true + - is_false: test.fields._source + - is_true: test.fields._recovery_source diff --git a/server/src/main/java/org/elasticsearch/action/get/TransportGetAction.java b/server/src/main/java/org/elasticsearch/action/get/TransportGetAction.java index d9caec599b8a7..ad1b4793c8c1b 100644 --- a/server/src/main/java/org/elasticsearch/action/get/TransportGetAction.java +++ b/server/src/main/java/org/elasticsearch/action/get/TransportGetAction.java @@ -101,7 +101,7 @@ protected void asyncShardOperation(GetRequest request, ShardId shardId, ActionLi } @Override - protected GetResponse shardOperation(GetRequest request, ShardId shardId) { + protected GetResponse shardOperation(GetRequest request, ShardId shardId) throws IOException { IndexService indexService = indicesService.indexServiceSafe(shardId.getIndex()); IndexShard indexShard = indexService.getShard(shardId.id()); diff --git a/server/src/main/java/org/elasticsearch/action/get/TransportShardMultiGetAction.java b/server/src/main/java/org/elasticsearch/action/get/TransportShardMultiGetAction.java index 77ea02db9bf2b..f443a05292efb 100644 --- a/server/src/main/java/org/elasticsearch/action/get/TransportShardMultiGetAction.java +++ b/server/src/main/java/org/elasticsearch/action/get/TransportShardMultiGetAction.java @@ -125,6 +125,9 @@ protected MultiGetShardResponse shardOperation(MultiGetShardRequest request, Sha logger.debug(() -> new ParameterizedMessage("{} failed to execute multi_get for [{}]", shardId, item.id()), e); response.add(request.locations.get(i), new MultiGetResponse.Failure(request.index(), item.id(), e)); } + } catch (IOException e) { + logger.debug(() -> new ParameterizedMessage("{} failed to execute multi_get for [{}]", shardId, item.id()), e); + response.add(request.locations.get(i), new MultiGetResponse.Failure(request.index(), item.id(), e)); } } diff --git a/server/src/main/java/org/elasticsearch/action/update/TransportUpdateAction.java b/server/src/main/java/org/elasticsearch/action/update/TransportUpdateAction.java index fee4ef047bea7..7c67bb045d846 100644 --- a/server/src/main/java/org/elasticsearch/action/update/TransportUpdateAction.java +++ b/server/src/main/java/org/elasticsearch/action/update/TransportUpdateAction.java @@ -173,10 +173,15 @@ protected ShardIterator shards(ClusterState clusterState, UpdateRequest request) @Override protected void shardOperation(final UpdateRequest request, final ActionListener listener) { - shardOperation(request, listener, 0); + try { + shardOperation(request, listener, 0); + } catch (IOException e) { + listener.onFailure(e); + } } - protected void shardOperation(final UpdateRequest request, final ActionListener listener, final int retryCount) { + protected void shardOperation(final UpdateRequest request, final ActionListener listener, final int retryCount) + throws IOException { final ShardId shardId = request.getShardId(); final IndexService indexService = indicesService.indexServiceSafe(shardId.getIndex()); final IndexShard indexShard = indexService.getShard(shardId.getId()); diff --git a/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java b/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java index 575ec265bbd73..06b850815681e 100644 --- a/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java +++ b/server/src/main/java/org/elasticsearch/action/update/UpdateHelper.java @@ -58,7 +58,7 @@ public UpdateHelper(ScriptService scriptService) { /** * Prepares an update request by converting it into an index or delete request or an update response (no action). */ - public Result prepare(UpdateRequest request, IndexShard indexShard, LongSupplier nowInMillis) { + public Result prepare(UpdateRequest request, IndexShard indexShard, LongSupplier nowInMillis) throws IOException { final GetResult getResult = indexShard.getService().getForUpdate(request.id(), request.ifSeqNo(), request.ifPrimaryTerm()); return prepare(indexShard.shardId(), request, getResult, nowInMillis); } diff --git a/server/src/main/java/org/elasticsearch/index/get/ShardGetService.java b/server/src/main/java/org/elasticsearch/index/get/ShardGetService.java index 4dfe6793672f3..b3e67530a8e49 100644 --- a/server/src/main/java/org/elasticsearch/index/get/ShardGetService.java +++ b/server/src/main/java/org/elasticsearch/index/get/ShardGetService.java @@ -71,7 +71,7 @@ public GetResult get( long version, VersionType versionType, FetchSourceContext fetchSourceContext - ) { + ) throws IOException { return get(id, gFields, realtime, version, versionType, UNASSIGNED_SEQ_NO, UNASSIGNED_PRIMARY_TERM, fetchSourceContext); } @@ -84,7 +84,7 @@ private GetResult get( long ifSeqNo, long ifPrimaryTerm, FetchSourceContext fetchSourceContext - ) { + ) throws IOException { currentMetric.inc(); try { long now = System.nanoTime(); @@ -101,7 +101,7 @@ private GetResult get( } } - public GetResult getForUpdate(String id, long ifSeqNo, long ifPrimaryTerm) { + public GetResult getForUpdate(String id, long ifSeqNo, long ifPrimaryTerm) throws IOException { return get( id, new String[] { RoutingFieldMapper.NAME }, @@ -121,7 +121,8 @@ public GetResult getForUpdate(String id, long ifSeqNo, long ifPrimaryTerm) { *

* Note: Call must release engine searcher associated with engineGetResult! */ - public GetResult get(Engine.GetResult engineGetResult, String id, String[] fields, FetchSourceContext fetchSourceContext) { + public GetResult get(Engine.GetResult engineGetResult, String id, String[] fields, FetchSourceContext fetchSourceContext) + throws IOException { if (engineGetResult.exists() == false) { return new GetResult(shardId.getIndexName(), id, UNASSIGNED_SEQ_NO, UNASSIGNED_PRIMARY_TERM, -1, false, null, null, null); } @@ -169,7 +170,7 @@ private GetResult innerGet( long ifSeqNo, long ifPrimaryTerm, FetchSourceContext fetchSourceContext - ) { + ) throws IOException { fetchSourceContext = normalizeFetchSourceContent(fetchSourceContext, gFields); Engine.GetResult get = indexShard.get( @@ -199,7 +200,7 @@ private GetResult innerGetLoadFromStoredFields( String[] storedFields, FetchSourceContext fetchSourceContext, Engine.GetResult get - ) { + ) throws IOException { assert get.exists() : "method should only be called if document could be retrieved"; // check first if stored fields to be loaded don't contain an object field @@ -227,7 +228,7 @@ private GetResult innerGetLoadFromStoredFields( } catch (IOException e) { throw new ElasticsearchException("Failed to get id [" + id + "]", e); } - source = fieldVisitor.source(); + source = mappingLookup.newSourceLoader().leaf(docIdAndVersion.reader).source(fieldVisitor, docIdAndVersion.docId); // put stored fields into result objects if (fieldVisitor.fields().isEmpty() == false) { diff --git a/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java index 80c2aeb35fcb9..7c238f9a75e12 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/BooleanFieldMapper.java @@ -424,4 +424,27 @@ public FieldMapper.Builder getMergeBuilder() { protected String contentType() { return CONTENT_TYPE; } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (hasScript()) { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } + if (hasDocValues == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + return new NumberFieldMapper.NumericSyntheticFieldLoader(name(), simpleName()) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(value == 1); + } + }; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java index 4a2431fc55c95..d5d601fb264c3 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java @@ -51,6 +51,7 @@ import org.elasticsearch.search.lookup.FieldValues; import org.elasticsearch.search.lookup.SearchLookup; import org.elasticsearch.search.runtime.LongScriptFieldDistanceFeatureQuery; +import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; import java.text.NumberFormat; @@ -869,4 +870,32 @@ public boolean getIgnoreMalformed() { public Long getNullValue() { return nullValue; } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (hasScript) { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } + if (hasDocValues == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (ignoreMalformed) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it ignores malformed dates" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + return new NumberFieldMapper.NumericSyntheticFieldLoader(name(), simpleName()) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(fieldType().format(value, fieldType().dateTimeFormatter())); + } + }; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java index c4e87edfeae94..46e51d92a399b 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/DocumentMapper.java @@ -90,6 +90,13 @@ public void validate(IndexSettings settings, boolean checkLimits) { ); } } + + /* + * Build an empty source loader to validate that the mapping is compatible + * with the source loading strategy declared on the source field mapper. + */ + sourceMapper().newSourceLoader(mapping().getRoot()); + settings.getMode().validateMapping(mappingLookup); if (settings.getIndexSortConfig().hasIndexSort() && mappers().nestedLookup() != NestedLookup.EMPTY) { throw new IllegalArgumentException("cannot have nested fields when index sort is activated"); diff --git a/server/src/main/java/org/elasticsearch/index/mapper/FieldAliasMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/FieldAliasMapper.java index 311c724659c3d..29eb960da27e6 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/FieldAliasMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/FieldAliasMapper.java @@ -156,4 +156,9 @@ public FieldAliasMapper build(MapperBuilderContext context) { return new FieldAliasMapper(name, fullName, path); } } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java index cd40ff1a3ce36..c4e8f39458669 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/GeoPointFieldMapper.java @@ -11,6 +11,7 @@ import org.apache.lucene.document.LatLonPoint; import org.apache.lucene.document.ShapeField; import org.apache.lucene.document.StoredField; +import org.apache.lucene.geo.GeoEncodingUtils; import org.apache.lucene.geo.LatLonGeometry; import org.apache.lucene.index.LeafReaderContext; import org.apache.lucene.search.IndexOrDocValuesQuery; @@ -40,6 +41,7 @@ import org.elasticsearch.search.lookup.SearchLookup; import org.elasticsearch.search.runtime.GeoPointScriptFieldDistanceFeatureQuery; import org.elasticsearch.xcontent.FilterXContentParserWrapper; +import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentParser; @@ -442,4 +444,35 @@ public GeoPoint normalizeFromSource(GeoPoint point) { return point; } } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (hasScript()) { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } + if (fieldType().hasDocValues() == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (ignoreMalformed()) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it ignores malformed points" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + return new NumberFieldMapper.NumericSyntheticFieldLoader(name(), simpleName()) { + final GeoPoint point = new GeoPoint(); + + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + point.reset(GeoEncodingUtils.decodeLatitude((int) (value >>> 32)), GeoEncodingUtils.decodeLongitude((int) value)); + point.toXContent(b, ToXContent.EMPTY_PARAMS); + } + }; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java index b2eb3b9979a15..625dadc138172 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/IpFieldMapper.java @@ -21,6 +21,7 @@ import org.elasticsearch.common.logging.DeprecationCategory; import org.elasticsearch.common.logging.DeprecationLogger; import org.elasticsearch.common.network.InetAddresses; +import org.elasticsearch.common.network.NetworkAddress; import org.elasticsearch.core.Nullable; import org.elasticsearch.core.Tuple; import org.elasticsearch.index.fielddata.IndexFieldData; @@ -34,11 +35,13 @@ import org.elasticsearch.search.aggregations.support.CoreValuesSourceType; import org.elasticsearch.search.lookup.FieldValues; import org.elasticsearch.search.lookup.SearchLookup; +import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentParser; import java.io.IOException; import java.net.InetAddress; import java.time.ZoneId; +import java.util.Arrays; import java.util.Collection; import java.util.Collections; import java.util.List; @@ -531,4 +534,33 @@ public void doValidate(MappingLookup lookup) { ); } } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (hasScript()) { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } + if (hasDocValues == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (ignoreMalformed) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it ignores malformed ips" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + return new KeywordFieldMapper.BytesSyntheticFieldLoader(name(), simpleName()) { + @Override + protected void loadNextValue(XContentBuilder b, BytesRef value) throws IOException { + byte[] bytes = Arrays.copyOfRange(value.bytes, value.offset, value.offset + value.length); + b.value(NetworkAddress.format(InetAddressPoint.decode(bytes))); + } + }; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java index d4b61aeaf0368..b0a13c68ad74e 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/KeywordFieldMapper.java @@ -16,9 +16,11 @@ import org.apache.lucene.document.Field; import org.apache.lucene.document.FieldType; import org.apache.lucene.document.SortedSetDocValuesField; +import org.apache.lucene.index.DocValues; import org.apache.lucene.index.FilteredTermsEnum; import org.apache.lucene.index.IndexOptions; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.LeafReader; import org.apache.lucene.index.LeafReaderContext; import org.apache.lucene.index.MultiTerms; import org.apache.lucene.index.ReaderSlice; @@ -61,6 +63,7 @@ import org.elasticsearch.search.runtime.StringScriptFieldRegexpQuery; import org.elasticsearch.search.runtime.StringScriptFieldTermQuery; import org.elasticsearch.search.runtime.StringScriptFieldWildcardQuery; +import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentParser; import java.io.IOException; @@ -1037,4 +1040,93 @@ public void doValidate(MappingLookup lookup) { ); } } + + boolean hasNormalizer() { + return normalizerName != null; + } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + return syntheticFieldLoader(simpleName()); + } + + protected SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String simpleName) { + if (hasScript()) { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } + if (hasDocValues == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (ignoreAbove != Defaults.IGNORE_ABOVE) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares ignore_above" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + if (hasNormalizer()) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares a normalizer" + ); + } + return new BytesSyntheticFieldLoader(name(), simpleName) { + @Override + protected void loadNextValue(XContentBuilder b, BytesRef value) throws IOException { + b.value(value.utf8ToString()); + } + }; + } + + public abstract static class BytesSyntheticFieldLoader implements SourceLoader.SyntheticFieldLoader { + private final String name; + private final String simpleName; + + public BytesSyntheticFieldLoader(String name, String simpleName) { + this.name = name; + this.simpleName = simpleName; + } + + @Override + public Leaf leaf(LeafReader reader) throws IOException { + SortedSetDocValues leaf = DocValues.getSortedSet(reader, name); + return new SourceLoader.SyntheticFieldLoader.Leaf() { + private boolean hasValue; + + @Override + public void advanceToDoc(int docId) throws IOException { + hasValue = leaf.advanceExact(docId); + } + + @Override + public boolean hasValue() { + return hasValue; + } + + @Override + public void load(XContentBuilder b) throws IOException { + long first = leaf.nextOrd(); + long next = leaf.nextOrd(); + if (next == SortedSetDocValues.NO_MORE_ORDS) { + b.field(simpleName); + loadNextValue(b, leaf.lookupOrd(first)); + return; + } + b.startArray(simpleName); + loadNextValue(b, leaf.lookupOrd(first)); + loadNextValue(b, leaf.lookupOrd(next)); + while ((next = leaf.nextOrd()) != SortedSetDocValues.NO_MORE_ORDS) { + loadNextValue(b, leaf.lookupOrd(next)); + } + b.endArray(); + } + }; + } + + protected abstract void loadNextValue(XContentBuilder b, BytesRef value) throws IOException; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java b/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java index 4fb82dfa065dc..724664e13a6f2 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/Mapper.java @@ -17,7 +17,6 @@ import java.util.Objects; public abstract class Mapper implements ToXContentFragment, Iterable { - public abstract static class Builder { protected final String name; @@ -76,6 +75,18 @@ public final String simpleName() { */ public abstract void validate(MappingLookup mappers); + /** + * Create a {@link SourceLoader.SyntheticFieldLoader} to populate synthetic source. + * + * @throws IllegalArgumentException if the field is configured in a way that doesn't + * support synthetic source. This translates nicely into a 400 error when + * users configure synthetic source in the mapping without configuring all + * fields properly. + */ + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + throw new IllegalArgumentException("field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source"); + } + @Override public String toString() { return Strings.toString(this); diff --git a/server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java b/server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java index 287978f11da52..e5f1eddc5c00e 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java @@ -403,6 +403,11 @@ public boolean isSourceEnabled() { return sfm != null && sfm.enabled(); } + public SourceLoader newSourceLoader() { + SourceFieldMapper sfm = mapping.getMetadataMapperByClass(SourceFieldMapper.class); + return sfm == null ? SourceLoader.FROM_STORED_SOURCE : sfm.newSourceLoader(mapping.getRoot()); + } + /** * Returns if this mapping contains a data-stream's timestamp meta-field and this field is enabled. * Only indices that are a part of a data-stream have this meta-field enabled. diff --git a/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java index 584fdf586f3b6..56f053812f1c6 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/NestedObjectMapper.java @@ -191,4 +191,9 @@ public ObjectMapper merge(Mapper mergeWith, MapperService.MergeReason reason) { toMerge.doMerge(mergeWithObject, reason); return toMerge; } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + throw new IllegalArgumentException("field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source"); + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java index 631c63bb497b5..307efc31000b0 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java @@ -15,7 +15,10 @@ import org.apache.lucene.document.LongPoint; import org.apache.lucene.document.SortedNumericDocValuesField; import org.apache.lucene.document.StoredField; +import org.apache.lucene.index.DocValues; +import org.apache.lucene.index.LeafReader; import org.apache.lucene.index.LeafReaderContext; +import org.apache.lucene.index.SortedNumericDocValues; import org.apache.lucene.sandbox.document.HalfFloatPoint; import org.apache.lucene.sandbox.search.IndexSortSortedNumericDocValuesRangeQuery; import org.apache.lucene.search.IndexOrDocValuesQuery; @@ -375,6 +378,16 @@ private static void validateParsed(float value) { throw new IllegalArgumentException("[half_float] supports only finite values, but got [" + value + "]"); } } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return new NumericSyntheticFieldLoader(fieldName, fieldSimpleName) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(HalfFloatPoint.sortableShortToHalfFloat((short) value)); + } + }; + } }, FLOAT("float", NumericType.FLOAT) { @Override @@ -501,6 +514,16 @@ private static void validateParsed(float value) { throw new IllegalArgumentException("[float] supports only finite values, but got [" + value + "]"); } } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return new NumericSyntheticFieldLoader(fieldName, fieldSimpleName) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(NumericUtils.sortableIntToFloat((int) value)); + } + }; + } }, DOUBLE("double", NumericType.DOUBLE) { @Override @@ -605,6 +628,16 @@ private static void validateParsed(double value) { throw new IllegalArgumentException("[double] supports only finite values, but got [" + value + "]"); } } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return new NumericSyntheticFieldLoader(fieldName, fieldSimpleName) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(NumericUtils.sortableLongToDouble(value)); + } + }; + } }, BYTE("byte", NumericType.BYTE) { @Override @@ -677,6 +710,11 @@ Number valueForSearch(Number value) { public IndexFieldData.Builder getFieldDataBuilder(String name) { return new SortedNumericIndexFieldData.Builder(name, numericType(), ByteDocValuesField::new); } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return NumberType.syntheticLongFieldLoader(fieldName, fieldSimpleName); + } }, SHORT("short", NumericType.SHORT) { @Override @@ -745,6 +783,11 @@ Number valueForSearch(Number value) { public IndexFieldData.Builder getFieldDataBuilder(String name) { return new SortedNumericIndexFieldData.Builder(name, numericType(), ShortDocValuesField::new); } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return NumberType.syntheticLongFieldLoader(fieldName, fieldSimpleName); + } }, INTEGER("integer", NumericType.INT) { @Override @@ -881,6 +924,11 @@ public List createFields(String name, Number value, boolean indexed, bool public IndexFieldData.Builder getFieldDataBuilder(String name) { return new SortedNumericIndexFieldData.Builder(name, numericType(), IntegerDocValuesField::new); } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return NumberType.syntheticLongFieldLoader(fieldName, fieldSimpleName); + } }, LONG("long", NumericType.LONG) { @Override @@ -987,6 +1035,11 @@ public List createFields(String name, Number value, boolean indexed, bool public IndexFieldData.Builder getFieldDataBuilder(String name) { return new SortedNumericIndexFieldData.Builder(name, numericType(), LongDocValuesField::new); } + + @Override + SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName) { + return syntheticLongFieldLoader(fieldName, fieldSimpleName); + } }; private final String name; @@ -1201,6 +1254,17 @@ public static Query longRangeQuery( public double reduceToStoredPrecision(double value) { return ((Number) value).doubleValue(); } + + abstract SourceLoader.SyntheticFieldLoader syntheticFieldLoader(String fieldName, String fieldSimpleName); + + private static SourceLoader.SyntheticFieldLoader syntheticLongFieldLoader(String fieldName, String fieldSimpleName) { + return new NumericSyntheticFieldLoader(fieldName, fieldSimpleName) { + @Override + protected void loadNextValue(XContentBuilder b, long value) throws IOException { + b.value(value); + } + }; + } } public static class NumberFieldType extends SimpleMappedFieldType { @@ -1523,4 +1587,71 @@ public void doValidate(MappingLookup lookup) { ); } } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (hasScript()) { + return SourceLoader.SyntheticFieldLoader.NOTHING; + } + if (hasDocValues == false) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it doesn't have doc values" + ); + } + if (ignoreMalformed.value()) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it ignores malformed numbers" + ); + } + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + return type.syntheticFieldLoader(name(), simpleName()); + } + + public abstract static class NumericSyntheticFieldLoader implements SourceLoader.SyntheticFieldLoader { + private final String name; + private final String simpleName; + + protected NumericSyntheticFieldLoader(String name, String simpleName) { + this.name = name; + this.simpleName = simpleName; + } + + @Override + public Leaf leaf(LeafReader reader) throws IOException { + SortedNumericDocValues leaf = DocValues.getSortedNumeric(reader, name); + return new SourceLoader.SyntheticFieldLoader.Leaf() { + private boolean hasValue; + + @Override + public void advanceToDoc(int docId) throws IOException { + hasValue = leaf.advanceExact(docId); + } + + @Override + public boolean hasValue() { + return hasValue; + } + + @Override + public void load(XContentBuilder b) throws IOException { + if (leaf.docValueCount() == 1) { + b.field(simpleName); + loadNextValue(b, leaf.nextValue()); + return; + } + b.startArray(simpleName); + for (int i = 0; i < leaf.docValueCount(); i++) { + loadNextValue(b, leaf.nextValue()); + } + b.endArray(); + } + }; + } + + protected abstract void loadNextValue(XContentBuilder b, long value) throws IOException; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java index e0753bd1c8002..9ec4559617612 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/ObjectMapper.java @@ -8,6 +8,7 @@ package org.elasticsearch.index.mapper; +import org.apache.lucene.index.LeafReader; import org.elasticsearch.ElasticsearchParseException; import org.elasticsearch.Version; import org.elasticsearch.common.Explicit; @@ -495,4 +496,63 @@ protected void serializeMappers(XContentBuilder builder, Params params) throws I protected void doXContent(XContentBuilder builder, Params params) throws IOException { } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + List fields = new ArrayList<>(); + mappers.values().stream().sorted(Comparator.comparing(Mapper::name)).forEach(sub -> { + SourceLoader.SyntheticFieldLoader subLoader = sub.syntheticFieldLoader(); + if (subLoader != null) { + fields.add(subLoader); + } + }); + return new SourceLoader.SyntheticFieldLoader() { + @Override + public Leaf leaf(LeafReader reader) throws IOException { + List leaves = new ArrayList<>(); + for (SourceLoader.SyntheticFieldLoader field : fields) { + leaves.add(field.leaf(reader)); + } + return new SourceLoader.SyntheticFieldLoader.Leaf() { + @Override + public void advanceToDoc(int docId) throws IOException { + for (SourceLoader.SyntheticFieldLoader.Leaf leaf : leaves) { + leaf.advanceToDoc(docId); + } + } + + @Override + public boolean hasValue() { + for (SourceLoader.SyntheticFieldLoader.Leaf leaf : leaves) { + if (leaf.hasValue()) { + return true; + } + } + return false; + } + + @Override + public void load(XContentBuilder b) throws IOException { + boolean started = false; + for (SourceLoader.SyntheticFieldLoader.Leaf leaf : leaves) { + if (leaf.hasValue()) { + if (false == started) { + started = true; + startSyntheticField(b); + } + leaf.load(b); + } + } + if (started) { + b.endObject(); + } + } + }; + } + }; + } + + protected void startSyntheticField(XContentBuilder b) throws IOException { + b.startObject(simpleName()); + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java index 11867911eb36d..4ecc39269592c 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java @@ -508,4 +508,9 @@ private static boolean containsSnippet(Object value, String snippet) { } return false; } + + @Override + protected void startSyntheticField(XContentBuilder b) throws IOException { + b.startObject(); + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/SourceFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/SourceFieldMapper.java index f618e131e4d22..6f90397204b5c 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/SourceFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/SourceFieldMapper.java @@ -19,6 +19,7 @@ import org.elasticsearch.common.util.CollectionUtils; import org.elasticsearch.common.xcontent.XContentFieldFilter; import org.elasticsearch.core.Nullable; +import org.elasticsearch.index.IndexSettings; import org.elasticsearch.index.query.QueryShardException; import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.xcontent.XContentType; @@ -35,11 +36,17 @@ public class SourceFieldMapper extends MetadataFieldMapper { public static final String CONTENT_TYPE = "_source"; private final XContentFieldFilter filter; - private static final SourceFieldMapper DEFAULT = new SourceFieldMapper(Defaults.ENABLED, Strings.EMPTY_ARRAY, Strings.EMPTY_ARRAY); + private static final SourceFieldMapper DEFAULT = new SourceFieldMapper( + Defaults.ENABLED, + Defaults.SYNTHETIC, + Strings.EMPTY_ARRAY, + Strings.EMPTY_ARRAY + ); public static class Defaults { public static final String NAME = SourceFieldMapper.NAME; public static final boolean ENABLED = true; + public static final boolean SYNTHETIC = false; public static final FieldType FIELD_TYPE = new FieldType(); @@ -60,6 +67,7 @@ public static class Builder extends MetadataFieldMapper.Builder { private final Parameter enabled = Parameter.boolParam("enabled", false, m -> toType(m).enabled, Defaults.ENABLED) // this field mapper may be enabled but once enabled, may not be disabled .setMergeValidator((previous, current, conflicts) -> (previous == current) || (previous && current == false)); + private final Parameter synthetic = Parameter.boolParam("synthetic", false, m -> toType(m).synthetic, false); private final Parameter> includes = Parameter.stringArrayParam( "includes", false, @@ -77,16 +85,23 @@ public Builder() { @Override protected List> getParameters() { + if (IndexSettings.isTimeSeriesModeEnabled()) { + return List.of(enabled, synthetic, includes, excludes); + } return List.of(enabled, includes, excludes); } @Override public SourceFieldMapper build() { - if (enabled.getValue() == Defaults.ENABLED && includes.getValue().isEmpty() && excludes.getValue().isEmpty()) { + if (enabled.getValue() == Defaults.ENABLED + && synthetic.getValue() == Defaults.SYNTHETIC + && includes.getValue().isEmpty() + && excludes.getValue().isEmpty()) { return DEFAULT; } return new SourceFieldMapper( enabled.getValue(), + synthetic.getValue(), includes.getValue().toArray(String[]::new), excludes.getValue().toArray(String[]::new) ); @@ -125,20 +140,25 @@ public Query termQuery(Object value, SearchExecutionContext context) { private final boolean enabled; /** indicates whether the source will always exist and be complete, for use by features like the update API */ private final boolean complete; + private final boolean synthetic; private final String[] includes; private final String[] excludes; - private SourceFieldMapper(boolean enabled, String[] includes, String[] excludes) { + private SourceFieldMapper(boolean enabled, boolean synthetic, String[] includes, String[] excludes) { super(new SourceFieldType(enabled)); this.enabled = enabled; + this.synthetic = synthetic; this.includes = includes; this.excludes = excludes; final boolean filtered = CollectionUtils.isEmpty(includes) == false || CollectionUtils.isEmpty(excludes) == false; + if (filtered && synthetic) { + throw new IllegalArgumentException("filtering the stored _source is incompatible with synthetic source"); + } this.filter = enabled && filtered ? XContentFieldFilter.newFieldFilter(includes, excludes) : (sourceBytes, contentType) -> sourceBytes; - this.complete = enabled && CollectionUtils.isEmpty(includes) && CollectionUtils.isEmpty(excludes); + this.complete = enabled && synthetic == false && CollectionUtils.isEmpty(includes) && CollectionUtils.isEmpty(excludes); } public boolean enabled() { @@ -170,7 +190,7 @@ public void preParse(DocumentParserContext context) throws IOException { @Nullable public BytesReference applyFilters(@Nullable BytesReference originalSource, @Nullable XContentType contentType) throws IOException { - if (enabled && originalSource != null) { + if (enabled && synthetic == false && originalSource != null) { // Percolate and tv APIs may not set the source and that is ok, because these APIs will not index any data return filter.apply(originalSource, contentType); } else { @@ -187,4 +207,15 @@ protected String contentType() { public FieldMapper.Builder getMergeBuilder() { return new Builder().init(this); } + + public SourceLoader newSourceLoader(RootObjectMapper root) { + if (synthetic) { + return new SourceLoader.Synthetic(root); + } + return SourceLoader.FROM_STORED_SOURCE; + } + + public boolean isSynthetic() { + return synthetic; + } } diff --git a/server/src/main/java/org/elasticsearch/index/mapper/SourceLoader.java b/server/src/main/java/org/elasticsearch/index/mapper/SourceLoader.java new file mode 100644 index 0000000000000..3967cd50d2a3b --- /dev/null +++ b/server/src/main/java/org/elasticsearch/index/mapper/SourceLoader.java @@ -0,0 +1,92 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +package org.elasticsearch.index.mapper; + +import org.apache.lucene.index.LeafReader; +import org.elasticsearch.common.bytes.BytesReference; +import org.elasticsearch.index.fieldvisitor.FieldsVisitor; +import org.elasticsearch.xcontent.XContentBuilder; +import org.elasticsearch.xcontent.json.JsonXContent; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; + +public interface SourceLoader { + interface Leaf { + BytesReference source(FieldsVisitor fieldsVisitor, int docId) throws IOException; + } + + Leaf leaf(LeafReader reader) throws IOException; + + SourceLoader FROM_STORED_SOURCE = new SourceLoader() { + @Override + public Leaf leaf(LeafReader reader) { + return new Leaf() { + @Override + public BytesReference source(FieldsVisitor fieldsVisitor, int docId) { + return fieldsVisitor.source(); + } + }; + } + }; + + class Synthetic implements SourceLoader { + private final SyntheticFieldLoader loader; + + Synthetic(RootObjectMapper root) { + loader = root.syntheticFieldLoader(); + } + + @Override + public Leaf leaf(LeafReader reader) throws IOException { + SyntheticFieldLoader.Leaf leaf = loader.leaf(reader); + return new Leaf() { + @Override + public BytesReference source(FieldsVisitor fieldsVisitor, int docId) throws IOException { + // TODO accept a requested xcontent type + try (XContentBuilder b = new XContentBuilder(JsonXContent.jsonXContent, new ByteArrayOutputStream())) { + leaf.advanceToDoc(docId); + if (leaf.hasValue()) { + leaf.load(b); + } else { + b.startObject().endObject(); + } + return BytesReference.bytes(b); + } + } + }; + } + } + + interface SyntheticFieldLoader { + SyntheticFieldLoader NOTHING = r -> new Leaf() { + @Override + public void advanceToDoc(int docId) throws IOException {} + + @Override + public boolean hasValue() { + return false; + } + + @Override + public void load(XContentBuilder b) throws IOException {} + }; + + Leaf leaf(LeafReader reader) throws IOException; + + interface Leaf { + void advanceToDoc(int docId) throws IOException; + + boolean hasValue(); + + void load(XContentBuilder b) throws IOException; + } + } + +} diff --git a/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java b/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java index f0b8b6de41493..fd9fbfd3e3505 100644 --- a/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java +++ b/server/src/main/java/org/elasticsearch/index/mapper/TextFieldMapper.java @@ -72,6 +72,7 @@ import java.util.Collections; import java.util.HashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Objects; import java.util.function.IntPredicate; @@ -1135,4 +1136,33 @@ protected void doXContentBody(XContentBuilder builder, Params params) throws IOE b.indexPrefixes.toXContent(builder, includeDefaults); b.indexPhrases.toXContent(builder, includeDefaults); } + + @Override + public SourceLoader.SyntheticFieldLoader syntheticFieldLoader() { + if (copyTo.copyToFields().isEmpty() != true) { + throw new IllegalArgumentException( + "field [" + name() + "] of type [" + typeName() + "] doesn't support synthetic source because it declares copy_to" + ); + } + for (Mapper sub : this) { + if (sub.typeName().equals(KeywordFieldMapper.CONTENT_TYPE)) { + KeywordFieldMapper kwd = (KeywordFieldMapper) sub; + if (kwd.fieldType().hasDocValues() + && kwd.hasNormalizer() == false + && kwd.fieldType().ignoreAbove() == KeywordFieldMapper.Defaults.IGNORE_ABOVE) { + + return kwd.syntheticFieldLoader(simpleName()); + } + } + } + throw new IllegalArgumentException( + String.format( + Locale.ROOT, + "field [%s] of type [%s] doesn't support synthetic source unless it has a sub-field of" + + " type [keyword] with doc values enabled and without ignore_above or a normalizer", + name(), + typeName() + ) + ); + } } diff --git a/server/src/main/java/org/elasticsearch/index/query/SearchExecutionContext.java b/server/src/main/java/org/elasticsearch/index/query/SearchExecutionContext.java index 80094f97346dc..8aef8b17807fd 100644 --- a/server/src/main/java/org/elasticsearch/index/query/SearchExecutionContext.java +++ b/server/src/main/java/org/elasticsearch/index/query/SearchExecutionContext.java @@ -46,6 +46,7 @@ import org.elasticsearch.index.mapper.NestedLookup; import org.elasticsearch.index.mapper.ParsedDocument; import org.elasticsearch.index.mapper.RuntimeField; +import org.elasticsearch.index.mapper.SourceLoader; import org.elasticsearch.index.mapper.SourceToParse; import org.elasticsearch.index.mapper.TextFieldMapper; import org.elasticsearch.index.query.support.NestedScope; @@ -389,6 +390,10 @@ public boolean isSourceEnabled() { return mappingLookup.isSourceEnabled(); } + public SourceLoader newSourceLoader() { + return mappingLookup.newSourceLoader(); + } + /** * Given a type (eg. long, string, ...), returns an anonymous field type that can be used for search operations. * Generally used to handle unmapped fields in the context of sorting. diff --git a/server/src/main/java/org/elasticsearch/search/fetch/FetchPhase.java b/server/src/main/java/org/elasticsearch/search/fetch/FetchPhase.java index 542456f401b07..c9ca3d7530e08 100644 --- a/server/src/main/java/org/elasticsearch/search/fetch/FetchPhase.java +++ b/server/src/main/java/org/elasticsearch/search/fetch/FetchPhase.java @@ -14,6 +14,7 @@ import org.apache.lucene.index.ReaderUtil; import org.apache.lucene.search.TotalHits; import org.elasticsearch.common.CheckedBiConsumer; +import org.elasticsearch.common.bytes.BytesReference; import org.elasticsearch.common.document.DocumentField; import org.elasticsearch.common.lucene.index.SequentialStoredFieldsLeafReader; import org.elasticsearch.common.xcontent.XContentHelper; @@ -23,6 +24,7 @@ import org.elasticsearch.index.fieldvisitor.FieldsVisitor; import org.elasticsearch.index.mapper.MappedFieldType; import org.elasticsearch.index.mapper.SourceFieldMapper; +import org.elasticsearch.index.mapper.SourceLoader; import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.search.LeafNestedDocuments; import org.elasticsearch.search.NestedDocuments; @@ -123,6 +125,8 @@ private SearchHits buildSearchHits(SearchContext context, Profiler profiler) { LeafNestedDocuments leafNestedDocuments = null; CheckedBiConsumer fieldReader = null; boolean hasSequentialDocs = hasSequentialDocs(docs); + SourceLoader sourceLoader = context.getSearchExecutionContext().newSourceLoader(); + SourceLoader.Leaf leafSourceLoader = null; for (int index = 0; index < context.docIdsToLoadSize(); index++) { if (context.isCancelled()) { throw new TaskCancelledException("cancelled"); @@ -146,6 +150,7 @@ private SearchHits buildSearchHits(SearchContext context, Profiler profiler) { } else { fieldReader = currentReaderContext.reader()::document; } + leafSourceLoader = sourceLoader.leaf(currentReaderContext.reader()); for (FetchSubPhaseProcessor processor : processors) { processor.setNextReader(currentReaderContext); } @@ -163,6 +168,7 @@ private SearchHits buildSearchHits(SearchContext context, Profiler profiler) { docId, storedToRequestedFields, currentReaderContext, + leafSourceLoader, fieldReader ); for (FetchSubPhaseProcessor processor : processors) { @@ -264,6 +270,7 @@ private static HitContext prepareHitContext( int docId, Map> storedToRequestedFields, LeafReaderContext subReaderContext, + SourceLoader.Leaf sourceLoader, CheckedBiConsumer storedFieldReader ) throws IOException { if (nestedDocuments.advance(docId - subReaderContext.docBase) == null) { @@ -274,6 +281,7 @@ private static HitContext prepareHitContext( docId, storedToRequestedFields, subReaderContext, + sourceLoader, storedFieldReader ); } else { @@ -303,6 +311,7 @@ private static HitContext prepareNonNestedHitContext( int docId, Map> storedToRequestedFields, LeafReaderContext subReaderContext, + SourceLoader.Leaf sourceLoader, CheckedBiConsumer fieldReader ) throws IOException { int subDocId = docId - subReaderContext.docBase; @@ -316,20 +325,32 @@ private static HitContext prepareNonNestedHitContext( Map docFields = new HashMap<>(); Map metaFields = new HashMap<>(); fillDocAndMetaFields(context, fieldsVisitor, storedToRequestedFields, docFields, metaFields); + hit = new SearchHit(docId, fieldsVisitor.id(), docFields, metaFields); } else { hit = new SearchHit(docId, fieldsVisitor.id(), emptyMap(), emptyMap()); } HitContext hitContext = new HitContext(hit, subReaderContext, subDocId); - if (fieldsVisitor.source() != null) { + BytesReference source; + if (sourceRequired(context)) { + try { + profiler.startLoadingSource(); + source = sourceLoader.source(fieldsVisitor, subDocId); + } finally { + profiler.stopLoadingSource(); + } + } else { + source = null; + } + if (source != null) { // Store the loaded source on the hit context so that fetch subphases can access it. // Also make it available to scripts by storing it on the shared SearchLookup instance. - hitContext.sourceLookup().setSource(fieldsVisitor.source()); + hitContext.sourceLookup().setSource(source); SourceLookup scriptSourceLookup = context.getSearchExecutionContext().lookup().source(); scriptSourceLookup.setSegmentAndDocument(subReaderContext, subDocId); - scriptSourceLookup.setSource(fieldsVisitor.source()); + scriptSourceLookup.setSource(source); } return hitContext; } @@ -498,6 +519,10 @@ interface Profiler { void stopLoadingStoredFields(); + void startLoadingSource(); + + void stopLoadingSource(); + void startNextReader(); void stopNextReader(); @@ -522,6 +547,12 @@ public void startLoadingStoredFields() {} @Override public void stopLoadingStoredFields() {} + @Override + public void startLoadingSource() {} + + @Override + public void stopLoadingSource() {} + @Override public void startNextReader() {} diff --git a/server/src/main/java/org/elasticsearch/search/fetch/FetchProfiler.java b/server/src/main/java/org/elasticsearch/search/fetch/FetchProfiler.java index 2f3670d904595..595a1d8bc2c62 100644 --- a/server/src/main/java/org/elasticsearch/search/fetch/FetchProfiler.java +++ b/server/src/main/java/org/elasticsearch/search/fetch/FetchProfiler.java @@ -104,6 +104,16 @@ public void stopLoadingStoredFields() { current.getTimer(FetchPhaseTiming.LOAD_STORED_FIELDS).stop(); } + @Override + public void startLoadingSource() { + current.getTimer(FetchPhaseTiming.LOAD_SOURCE).start(); + } + + @Override + public void stopLoadingSource() { + current.getTimer(FetchPhaseTiming.LOAD_SOURCE).stop(); + } + @Override public void startNextReader() { current.getTimer(FetchPhaseTiming.NEXT_READER).start(); @@ -138,9 +148,28 @@ ProfileResult result(long stop) { } } + /** + * Actions within the "main" fetch phase that are explicitly profiled. + * See also {@link FetchSubPhaseProfileBreakdown}. + */ enum FetchPhaseTiming { + /** + * Time spent setting up infrastructure for each segment. This is + * called once per segment that has a matching document. + */ NEXT_READER, - LOAD_STORED_FIELDS; + /** + * Time spent loading stored fields for each document. This is called + * once per document if the fetch needs stored fields. Most do. + */ + LOAD_STORED_FIELDS, + /** + * Time spent computing the {@code _source}. This is called once per + * document that needs to fetch source. This may be as fast as reading + * {@code _source} from the stored fields or as slow as loading doc + * values for all fields. + */ + LOAD_SOURCE; @Override public String toString() { @@ -148,6 +177,9 @@ public String toString() { } } + /** + * Timings from an optional sub-phase of fetch. + */ static class FetchSubPhaseProfileBreakdown extends AbstractProfileBreakdown { private final String type; private final String description; diff --git a/server/src/test/java/org/elasticsearch/index/mapper/BinaryFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/BinaryFieldMapperTests.java index 3808a3894a239..ac6f0d863e461 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/BinaryFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/BinaryFieldMapperTests.java @@ -16,6 +16,7 @@ import org.elasticsearch.common.compress.CompressorFactory; import org.elasticsearch.common.io.stream.BytesStreamOutput; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.io.OutputStream; @@ -136,4 +137,14 @@ protected void randomFetchTestFieldConfig(XContentBuilder b) throws IOException protected boolean dedupAfterFetch() { return true; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/BooleanFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/BooleanFieldMapperTests.java index fd1ce38dd55bf..078d1024e1749 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/BooleanFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/BooleanFieldMapperTests.java @@ -14,11 +14,14 @@ import org.apache.lucene.index.SortedNumericDocValues; import org.apache.lucene.util.BytesRef; import org.elasticsearch.common.Strings; +import org.elasticsearch.script.BooleanFieldScript; +import org.elasticsearch.test.ESTestCase; import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import java.io.IOException; +import java.util.List; import static org.hamcrest.Matchers.equalTo; @@ -190,4 +193,72 @@ public void testScriptAndPrecludedParameters() { }))); assertThat(e.getMessage(), equalTo("Failed to parse mapping: Field [null_value] cannot be set in conjunction with field [script]")); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new SyntheticSourceSupport() { + @Override + public SyntheticSourceExample example() throws IOException { + switch (randomInt(3)) { + case 0: + boolean v = randomBoolean(); + return new SyntheticSourceExample(v, v, BooleanFieldMapperTests.this::minimalMapping); + case 1: + List in = randomList(1, 5, ESTestCase::randomBoolean); + Object out = in.size() == 1 ? in.get(0) : in.stream().sorted().toList(); + return new SyntheticSourceExample(in, out, BooleanFieldMapperTests.this::minimalMapping); + case 2: + v = randomBoolean(); + return new SyntheticSourceExample(null, v, b -> { + minimalMapping(b); + b.field("null_value", v); + }); + case 3: + boolean nullValue = randomBoolean(); + List vals = randomList(1, 5, ESTestCase::randomBoolean); + in = vals.stream().map(b -> b == nullValue ? null : b).toList(); + out = vals.size() == 1 ? vals.get(0) : vals.stream().sorted().toList(); + return new SyntheticSourceExample(in, out, b -> { + minimalMapping(b); + b.field("null_value", nullValue); + }); + default: + throw new IllegalArgumentException(); + } + } + + @Override + public List invalidExample() throws IOException { + return List.of( + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [boolean] doesn't support synthetic source because it doesn't have doc values"), + b -> b.field("type", "boolean").field("doc_values", false) + ) + // If boolean had ignore_malformed we'd fail to index here + ); + } + }; + } + + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + protected BooleanFieldScript.Factory emptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new BooleanFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() {} + }; + } + + @Override + protected BooleanFieldScript.Factory nonEmptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new BooleanFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() { + emit(true); + } + }; + } + }; + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/ByteFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/ByteFieldMapperTests.java index 3052214d4a745..e1c4043f42963 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/ByteFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/ByteFieldMapperTests.java @@ -11,6 +11,7 @@ import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.List; @@ -46,4 +47,9 @@ protected Number randomNumber() { } return randomDoubleBetween(Byte.MIN_VALUE, Byte.MAX_VALUE, true); } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/CompletionFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/CompletionFieldMapperTests.java index a5935b7625280..eb987a30f966e 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/CompletionFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/CompletionFieldMapperTests.java @@ -47,6 +47,7 @@ import org.hamcrest.Matcher; import org.hamcrest.Matchers; import org.hamcrest.core.CombinableMatcher; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.ArrayList; @@ -943,4 +944,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("We don't have doc values or fielddata", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/DateFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/DateFieldMapperTests.java index 00c01cb23608e..78dce7b316f55 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/DateFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/DateFieldMapperTests.java @@ -14,16 +14,21 @@ import org.elasticsearch.common.settings.Settings; import org.elasticsearch.common.time.DateFormatter; import org.elasticsearch.common.time.DateUtils; +import org.elasticsearch.core.Tuple; import org.elasticsearch.index.mapper.DateFieldMapper.DateFieldType; import org.elasticsearch.index.termvectors.TermVectorsService; +import org.elasticsearch.script.DateFieldScript; import org.elasticsearch.search.DocValueFormat; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; import java.math.BigDecimal; +import java.time.Instant; import java.time.ZoneId; import java.time.ZoneOffset; import java.time.ZonedDateTime; +import java.util.ArrayList; +import java.util.Comparator; import java.util.List; import java.util.Locale; @@ -563,6 +568,141 @@ public void testScriptAndPrecludedParameters() { } } + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new SyntheticSourceSupport() { + private final DateFieldMapper.Resolution resolution = randomFrom(DateFieldMapper.Resolution.values()); + private final Object nullValue = usually() + ? null + : randomValueOtherThanMany( + v -> v instanceof BigDecimal, // BigDecimal values don't parse properly so limit the test to others + () -> randomValue() + ); + private final DateFormatter formatter = resolution == DateFieldMapper.Resolution.MILLISECONDS + ? DateFieldMapper.DEFAULT_DATE_TIME_FORMATTER + : DateFieldMapper.DEFAULT_DATE_TIME_NANOS_FORMATTER; + + @Override + public SyntheticSourceExample example() { + if (randomBoolean()) { + Tuple v = generateValue(); + return new SyntheticSourceExample(v.v1(), v.v2(), this::mapping); + } + List> values = randomList(1, 5, this::generateValue); + List in = values.stream().map(Tuple::v1).toList(); + List outList = values.stream() + .sorted( + Comparator.comparing(v -> Instant.from(formatter.parse(v.v1() == null ? nullValue.toString() : v.v1().toString()))) + ) + .map(Tuple::v2) + .toList(); + Object out = outList.size() == 1 ? outList.get(0) : outList; + return new SyntheticSourceExample(in, out, this::mapping); + } + + private Tuple generateValue() { + if (nullValue != null && randomBoolean()) { + return Tuple.tuple(null, outValue(nullValue)); + } + Object in = randomValue(); + String out = outValue(in); + return Tuple.tuple(in, out); + } + + private Object randomValue() { + switch (resolution) { + case MILLISECONDS: + if (randomBoolean()) { + return randomIs8601Nanos(MAX_ISO_DATE); + } + return randomLongBetween(0, MAX_ISO_DATE); + case NANOSECONDS: + return switch (randomInt(2)) { + case 0 -> randomLongBetween(0, MAX_NANOS); + case 1 -> randomIs8601Nanos(MAX_NANOS); + case 2 -> new BigDecimal(randomDecimalNanos(MAX_MILLIS_DOUBLE_NANOS_KEEPS_PRECISION)); + default -> throw new IllegalStateException(); + }; + default: + throw new IllegalStateException(); + } + } + + private String outValue(Object in) { + return formatter.format(formatter.parse(in.toString())); + } + + private void mapping(XContentBuilder b) throws IOException { + b.field("type", resolution.type()); + if (nullValue != null) { + b.field("null_value", nullValue); + } + } + + @Override + public List invalidExample() throws IOException { + List examples = new ArrayList<>(); + for (String fieldType : new String[] { "date", "date_nanos" }) { + examples.add( + new SyntheticSourceInvalidExample( + equalTo( + "field [field] of type [" + + fieldType + + "] doesn't support synthetic source because it doesn't have doc values" + ), + b -> b.field("type", fieldType).field("doc_values", false) + ) + ); + examples.add( + new SyntheticSourceInvalidExample( + equalTo( + "field [field] of type [" + + fieldType + + "] doesn't support synthetic source because it ignores malformed dates" + ), + b -> b.field("type", fieldType).field("ignore_malformed", true) + ) + ); + } + return examples; + } + }; + } + + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + protected DateFieldScript.Factory emptyFieldScript() { + return (fieldName, params, searchLookup, formatter) -> ctx -> new DateFieldScript( + fieldName, + params, + searchLookup, + formatter, + ctx + ) { + @Override + public void execute() {} + }; + } + + @Override + protected DateFieldScript.Factory nonEmptyFieldScript() { + return (fieldName, params, searchLookup, formatter) -> ctx -> new DateFieldScript( + fieldName, + params, + searchLookup, + formatter, + ctx + ) { + @Override + public void execute() { + emit(1649343081000L); + } + }; + } + }; + } + public void testLegacyField() throws Exception { // check that unknown date formats are treated leniently on old indices MapperService service = createMapperService(Version.fromString("5.0.0"), Settings.EMPTY, () -> false, mapping(b -> { diff --git a/server/src/test/java/org/elasticsearch/index/mapper/DateRangeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/DateRangeFieldMapperTests.java index 068c9575a2105..219b4dd2db6f9 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/DateRangeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/DateRangeFieldMapperTests.java @@ -9,6 +9,7 @@ package org.elasticsearch.index.mapper; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; @@ -53,4 +54,14 @@ public void testIllegalFormatField() { ); assertThat(e.getMessage(), containsString("Invalid format: [[test_format]]: Unknown pattern letter: t")); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/DoubleFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/DoubleFieldMapperTests.java index 5a30ea325e3b8..441430a21933d 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/DoubleFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/DoubleFieldMapperTests.java @@ -9,6 +9,9 @@ package org.elasticsearch.index.mapper; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; +import org.elasticsearch.script.DoubleFieldScript; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptContext; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; @@ -87,4 +90,41 @@ public void testScriptAndPrecludedParameters() { ); } } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + @SuppressWarnings("unchecked") + protected T compileOtherScript(Script script, ScriptContext context) { + if (context == DoubleFieldScript.CONTEXT) { + return (T) DoubleFieldScript.PARSE_FROM_SOURCE; + } + throw new UnsupportedOperationException("Unknown script " + script.getIdOrCode()); + } + + @Override + protected DoubleFieldScript.Factory emptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new DoubleFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() {} + }; + } + + @Override + protected DoubleFieldScript.Factory nonEmptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new DoubleFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() { + emit(1.0); + } + }; + } + }; + } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new NumberSyntheticSourceSupport(Number::doubleValue); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/DoubleRangeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/DoubleRangeFieldMapperTests.java index 2e82e856314ec..b7b1646b6b457 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/DoubleRangeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/DoubleRangeFieldMapperTests.java @@ -9,6 +9,7 @@ package org.elasticsearch.index.mapper; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; @@ -38,4 +39,14 @@ protected void minimalMapping(XContentBuilder b) throws IOException { protected boolean supportsDecimalCoerce() { return false; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/FloatFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/FloatFieldMapperTests.java index b3cfc85f7fb95..e77da451b9493 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/FloatFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/FloatFieldMapperTests.java @@ -10,6 +10,7 @@ import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.List; @@ -47,4 +48,14 @@ protected Number randomNumber() { */ return randomBoolean() ? randomDoubleBetween(-Float.MAX_VALUE, Float.MAX_VALUE, true) : randomFloat(); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new NumberSyntheticSourceSupport(Number::floatValue); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/FloatRangeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/FloatRangeFieldMapperTests.java index 62936a5ad196f..7385b0bf68db5 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/FloatRangeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/FloatRangeFieldMapperTests.java @@ -9,6 +9,7 @@ package org.elasticsearch.index.mapper; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; @@ -38,4 +39,14 @@ protected void minimalMapping(XContentBuilder b) throws IOException { protected boolean supportsDecimalCoerce() { return false; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/GeoPointFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/GeoPointFieldMapperTests.java index e152cacc71bd6..5b4afd19ba16d 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/GeoPointFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/GeoPointFieldMapperTests.java @@ -7,20 +7,30 @@ */ package org.elasticsearch.index.mapper; +import org.apache.lucene.document.LatLonDocValuesField; import org.apache.lucene.document.LatLonPoint; +import org.apache.lucene.geo.GeoEncodingUtils; import org.apache.lucene.index.IndexableField; import org.apache.lucene.util.BytesRef; import org.elasticsearch.common.Strings; import org.elasticsearch.common.geo.GeoPoint; +import org.elasticsearch.core.Tuple; +import org.elasticsearch.geo.GeometryTestUtils; +import org.elasticsearch.geometry.Point; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentParser; import org.elasticsearch.xcontent.XContentParserConfiguration; import org.elasticsearch.xcontent.json.JsonXContent; import org.hamcrest.CoreMatchers; +import org.junit.AssumptionViolatedException; import java.io.IOException; +import java.util.ArrayList; import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.TreeMap; import static org.elasticsearch.geometry.utils.Geohash.stringEncode; import static org.elasticsearch.test.ListMatcher.matchesList; @@ -456,4 +466,100 @@ public void testScriptAndPrecludedParameters() { ); } } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new SyntheticSourceSupport() { + private final boolean ignoreZValue = usually(); + private final GeoPoint nullValue = usually() ? null : randomGeoPoint(); + + @Override + public SyntheticSourceExample example() { + if (randomBoolean()) { + Tuple v = generateValue(); + return new SyntheticSourceExample(v.v1(), decode(encode(v.v2())), this::mapping); + } + List> values = randomList(1, 5, this::generateValue); + List in = values.stream().map(Tuple::v1).toList(); + List> outList = values.stream().map(t -> encode(t.v2())).sorted().map(this::decode).toList(); + Object out = outList.size() == 1 ? outList.get(0) : outList; + return new SyntheticSourceExample(in, out, this::mapping); + } + + private Tuple generateValue() { + if (nullValue != null && randomBoolean()) { + return Tuple.tuple(null, nullValue); + } + GeoPoint point = randomGeoPoint(); + return Tuple.tuple(randomGeoPointInput(point), point); + } + + private GeoPoint randomGeoPoint() { + Point point = GeometryTestUtils.randomPoint(false); + return new GeoPoint(point.getLat(), point.getLon()); + } + + private Object randomGeoPointInput(GeoPoint point) { + if (randomBoolean()) { + return Map.of("lat", point.lat(), "lon", point.lon()); + } + List coords = new ArrayList<>(); + coords.add(point.lon()); + coords.add(point.lat()); + if (ignoreZValue) { + coords.add(randomDouble()); + } + return Map.of("coordinates", coords, "type", "point"); + } + + private long encode(GeoPoint point) { + return new LatLonDocValuesField("f", point.lat(), point.lon()).numericValue().longValue(); + } + + private Map decode(long point) { + double lat = GeoEncodingUtils.decodeLatitude((int) (point >> 32)); + double lon = GeoEncodingUtils.decodeLongitude((int) (point & 0xFFFFFFFF)); + return new TreeMap<>(Map.of("lat", lat, "lon", lon)); + } + + private void mapping(XContentBuilder b) throws IOException { + b.field("type", "geo_point"); + if (ignoreZValue == false || rarely()) { + b.field("ignore_z_value", ignoreZValue); + } + if (nullValue != null) { + b.field("null_value", randomGeoPointInput(nullValue)); + } + if (rarely()) { + b.field("index", false); + } + if (rarely()) { + b.field("store", false); + } + } + + @Override + public List invalidExample() throws IOException { + return List.of( + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [geo_point] doesn't support synthetic source because it doesn't have doc values"), + b -> b.field("type", "geo_point").field("doc_values", false) + ), + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [geo_point] doesn't support synthetic source because it declares copy_to"), + b -> b.field("type", "geo_point").field("copy_to", "foo") + ), + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [geo_point] doesn't support synthetic source because it ignores malformed points"), + b -> b.field("type", "geo_point").field("ignore_malformed", true) + ) + ); + } + }; + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/GeoShapeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/GeoShapeFieldMapperTests.java index e4d8a0df4e64e..407a33b9ac776 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/GeoShapeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/GeoShapeFieldMapperTests.java @@ -13,6 +13,7 @@ import org.elasticsearch.test.TestGeoShapeFieldMapperPlugin; import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -221,4 +222,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/HalfFloatFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/HalfFloatFieldMapperTests.java index f23aa5e339c6d..73f46a85b3113 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/HalfFloatFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/HalfFloatFieldMapperTests.java @@ -8,9 +8,11 @@ package org.elasticsearch.index.mapper; +import org.apache.lucene.sandbox.document.HalfFloatPoint; import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.List; @@ -43,4 +45,16 @@ protected void minimalMapping(XContentBuilder b) throws IOException { protected Number randomNumber() { return randomBoolean() ? randomFloat() : randomDoubleBetween(-65504, 65504, true); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new NumberSyntheticSourceSupport( + n -> HalfFloatPoint.sortableShortToHalfFloat(HalfFloatPoint.halfFloatToSortableShort(n.floatValue())) + ); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/IntegerFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/IntegerFieldMapperTests.java index 657126a7bff97..994b74a25743c 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/IntegerFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/IntegerFieldMapperTests.java @@ -11,6 +11,7 @@ import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.List; @@ -47,4 +48,9 @@ protected Number randomNumber() { } return randomDoubleBetween(Integer.MIN_VALUE, Integer.MAX_VALUE, true); } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/IntegerRangeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/IntegerRangeFieldMapperTests.java index d2d4abb5ec80d..a8b61c0b0bd21 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/IntegerRangeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/IntegerRangeFieldMapperTests.java @@ -9,6 +9,7 @@ package org.elasticsearch.index.mapper; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; @@ -32,4 +33,14 @@ protected Object rangeValue() { protected void minimalMapping(XContentBuilder b) throws IOException { b.field("type", "integer_range"); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/IpFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/IpFieldMapperTests.java index 7ea421ab26455..189b7d59f063e 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/IpFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/IpFieldMapperTests.java @@ -18,11 +18,15 @@ import org.elasticsearch.Version; import org.elasticsearch.common.network.InetAddresses; import org.elasticsearch.common.network.NetworkAddress; +import org.elasticsearch.core.Tuple; import org.elasticsearch.index.termvectors.TermVectorsService; +import org.elasticsearch.script.IpFieldScript; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; import java.net.InetAddress; +import java.util.List; +import java.util.stream.Collectors; import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; @@ -297,4 +301,88 @@ public void testScriptAndPrecludedParameters() { ); } } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new SyntheticSourceSupport() { + private final InetAddress nullValue = usually() ? null : randomIp(randomBoolean()); + + @Override + public SyntheticSourceExample example() { + if (randomBoolean()) { + Tuple v = generateValue(); + return new SyntheticSourceExample(v.v1(), NetworkAddress.format(v.v2()), this::mapping); + } + List> values = randomList(1, 5, this::generateValue); + List in = values.stream().map(Tuple::v1).toList(); + List outList = values.stream() + .map(v -> new BytesRef(InetAddressPoint.encode(v.v2()))) + .collect(Collectors.toSet()) + .stream() + .sorted() + .map(v -> InetAddressPoint.decode(v.bytes)) + .map(NetworkAddress::format) + .toList(); + Object out = outList.size() == 1 ? outList.get(0) : outList; + return new SyntheticSourceExample(in, out, this::mapping); + } + + private Tuple generateValue() { + if (nullValue != null && randomBoolean()) { + return Tuple.tuple(null, nullValue); + } + InetAddress addr = randomIp(randomBoolean()); + return Tuple.tuple(NetworkAddress.format(addr), addr); + } + + private void mapping(XContentBuilder b) throws IOException { + b.field("type", "ip"); + if (nullValue != null) { + b.field("null_value", NetworkAddress.format(nullValue)); + } + if (rarely()) { + b.field("index", false); + } + if (rarely()) { + b.field("store", false); + } + } + + @Override + public List invalidExample() throws IOException { + return List.of( + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [ip] doesn't support synthetic source because it doesn't have doc values"), + b -> b.field("type", "ip").field("doc_values", false) + ), + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [ip] doesn't support synthetic source because it ignores malformed ips"), + b -> b.field("type", "ip").field("ignore_malformed", true) + ) + ); + } + }; + } + + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + protected IpFieldScript.Factory emptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new IpFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() {} + }; + } + + @Override + protected IpFieldScript.Factory nonEmptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new IpFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() { + emit("192.168.0.1"); + } + }; + } + }; + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/IpRangeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/IpRangeFieldMapperTests.java index b3b9e9ae653e9..e0f5eb3b47668 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/IpRangeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/IpRangeFieldMapperTests.java @@ -11,6 +11,7 @@ import org.apache.lucene.index.IndexableField; import org.elasticsearch.common.network.InetAddresses; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.HashMap; @@ -76,4 +77,14 @@ public void testStoreCidr() throws Exception { assertThat(storedField.stringValue(), containsString(strVal)); } } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/KeywordFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/KeywordFieldMapperTests.java index ef1591b834f7d..6f646cd59b3cf 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/KeywordFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/KeywordFieldMapperTests.java @@ -25,6 +25,7 @@ import org.elasticsearch.common.Strings; import org.elasticsearch.common.lucene.Lucene; import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.core.Tuple; import org.elasticsearch.index.IndexSettings; import org.elasticsearch.index.analysis.AnalyzerScope; import org.elasticsearch.index.analysis.CharFilterFactory; @@ -39,6 +40,7 @@ import org.elasticsearch.indices.analysis.AnalysisModule; import org.elasticsearch.plugins.AnalysisPlugin; import org.elasticsearch.plugins.Plugin; +import org.elasticsearch.script.StringFieldScript; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; @@ -46,6 +48,7 @@ import java.util.Collection; import java.util.List; import java.util.Map; +import java.util.stream.Collectors; import static java.util.Collections.singletonList; import static java.util.Collections.singletonMap; @@ -75,7 +78,6 @@ public Map> getTokeniz ) ); } - } @Override @@ -620,6 +622,84 @@ public void testKeywordFieldUtf8LongerThan32766() throws Exception { assertThat(e.getCause().getMessage(), containsString("UTF8 encoding is longer than the max length")); } + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new KeywordSyntheticSourceSupport(); + } + + static class KeywordSyntheticSourceSupport implements SyntheticSourceSupport { + private final String nullValue = usually() ? null : randomAlphaOfLength(2); + + @Override + public SyntheticSourceExample example() { + if (randomBoolean()) { + Tuple v = generateValue(); + return new SyntheticSourceExample(v.v1(), v.v2(), this::mapping); + } + List> values = randomList(1, 5, this::generateValue); + List in = values.stream().map(Tuple::v1).toList(); + List outList = values.stream().map(Tuple::v2).collect(Collectors.toSet()).stream().sorted().toList(); + Object out = outList.size() == 1 ? outList.get(0) : outList; + return new SyntheticSourceExample(in, out, this::mapping); + } + + private Tuple generateValue() { + if (nullValue != null && randomBoolean()) { + return Tuple.tuple(null, nullValue); + } + String v = randomAlphaOfLength(5); + return Tuple.tuple(v, v); + } + + private void mapping(XContentBuilder b) throws IOException { + b.field("type", "keyword"); + if (nullValue != null) { + b.field("null_value", nullValue); + } + } + + @Override + public List invalidExample() throws IOException { + return List.of( + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [keyword] doesn't support synthetic source because it doesn't have doc values"), + b -> b.field("type", "keyword").field("doc_values", false) + ), + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [keyword] doesn't support synthetic source because it declares ignore_above"), + b -> b.field("type", "keyword").field("ignore_above", 10) + ), + new SyntheticSourceInvalidExample( + equalTo("field [field] of type [keyword] doesn't support synthetic source because it declares a normalizer"), + b -> b.field("type", "keyword").field("normalizer", "lowercase") + ) + ); + } + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + protected StringFieldScript.Factory emptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new StringFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() {} + }; + } + + @Override + protected StringFieldScript.Factory nonEmptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new StringFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() { + emit("foo"); + } + }; + } + }; + } + public void testLegacyField() throws Exception { // check that unknown normalizers are treated leniently on old indices MapperService service = createMapperService(Version.fromString("5.0.0"), Settings.EMPTY, () -> false, mapping(b -> { diff --git a/server/src/test/java/org/elasticsearch/index/mapper/LongFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/LongFieldMapperTests.java index 1a415256b0b22..76264302114ba 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/LongFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/LongFieldMapperTests.java @@ -10,6 +10,9 @@ import org.elasticsearch.common.bytes.BytesArray; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; +import org.elasticsearch.script.LongFieldScript; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptContext; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentType; @@ -115,4 +118,35 @@ protected Number randomNumber() { public void testFetchCoerced() throws IOException { assertFetch(randomFetchTestMapper(), "field", 3.783147882954537E18, randomFetchTestFormat()); } + + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + @SuppressWarnings("unchecked") + protected T compileOtherScript(Script script, ScriptContext context) { + if (context == LongFieldScript.CONTEXT) { + return (T) LongFieldScript.PARSE_FROM_SOURCE; + } + throw new UnsupportedOperationException("Unknown script " + script.getIdOrCode()); + } + + @Override + protected LongFieldScript.Factory emptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new LongFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() {} + }; + } + + @Override + protected LongFieldScript.Factory nonEmptyFieldScript() { + return (fieldName, params, searchLookup) -> ctx -> new LongFieldScript(fieldName, params, searchLookup, ctx) { + @Override + public void execute() { + emit(1); + } + }; + } + }; + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/LongRangeFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/LongRangeFieldMapperTests.java index 643366f0c95da..102a9887012be 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/LongRangeFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/LongRangeFieldMapperTests.java @@ -9,6 +9,7 @@ package org.elasticsearch.index.mapper; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; @@ -33,4 +34,14 @@ protected Object rangeValue() { protected void minimalMapping(XContentBuilder b) throws IOException { b.field("type", "long_range"); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/NumberFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/NumberFieldMapperTests.java index 2c1bd847b394a..24342250faeda 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/NumberFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/NumberFieldMapperTests.java @@ -11,19 +11,23 @@ import org.apache.lucene.index.DocValuesType; import org.apache.lucene.index.IndexableField; import org.elasticsearch.common.Strings; +import org.elasticsearch.core.Tuple; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; import org.elasticsearch.index.termvectors.TermVectorsService; import org.elasticsearch.script.DoubleFieldScript; import org.elasticsearch.script.LongFieldScript; import org.elasticsearch.script.Script; import org.elasticsearch.script.ScriptContext; +import org.elasticsearch.script.ScriptFactory; import org.elasticsearch.xcontent.XContentBuilder; import java.io.IOException; import java.util.List; +import java.util.function.Function; import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; +import static org.hamcrest.Matchers.matchesPattern; public abstract class NumberFieldMapperTests extends MapperTestCase { @@ -303,15 +307,30 @@ protected final Object generateRandomInputValue(MappedFieldType ft) { } @Override - @SuppressWarnings("unchecked") - protected T compileScript(Script script, ScriptContext context) { - if (context == LongFieldScript.CONTEXT) { - return (T) LongFieldScript.PARSE_FROM_SOURCE; - } - if (context == DoubleFieldScript.CONTEXT) { - return (T) DoubleFieldScript.PARSE_FROM_SOURCE; - } - throw new UnsupportedOperationException("Unknown script " + script.getIdOrCode()); + protected IngestScriptSupport ingestScriptSupport() { + return new IngestScriptSupport() { + @Override + @SuppressWarnings("unchecked") + protected T compileOtherScript(Script script, ScriptContext context) { + if (context == LongFieldScript.CONTEXT) { + return (T) LongFieldScript.PARSE_FROM_SOURCE; + } + if (context == DoubleFieldScript.CONTEXT) { + return (T) DoubleFieldScript.PARSE_FROM_SOURCE; + } + throw new UnsupportedOperationException("Unknown script " + script.getIdOrCode()); + } + + @Override + ScriptFactory emptyFieldScript() { + return null; + } + + @Override + ScriptFactory nonEmptyFieldScript() { + return null; + } + }; } public void testScriptableTypes() throws IOException { @@ -331,4 +350,69 @@ public void testScriptableTypes() throws IOException { protected abstract Number randomNumber(); + protected final class NumberSyntheticSourceSupport implements SyntheticSourceSupport { + private final Function round; + private final Long nullValue = usually() ? null : randomNumber().longValue(); + private final boolean coerce = rarely(); + + protected NumberSyntheticSourceSupport(Function round) { + this.round = round; + } + + @Override + public SyntheticSourceExample example() { + if (randomBoolean()) { + Tuple v = generateValue(); + return new SyntheticSourceExample(v.v1(), round.apply(v.v2()), this::mapping); + } + List> values = randomList(1, 5, this::generateValue); + List in = values.stream().map(Tuple::v1).toList(); + List outList = values.stream().map(t -> round.apply(t.v2())).sorted().toList(); + Object out = outList.size() == 1 ? outList.get(0) : outList; + return new SyntheticSourceExample(in, out, this::mapping); + } + + private Tuple generateValue() { + if (nullValue != null && randomBoolean()) { + return Tuple.tuple(null, nullValue); + } + Number n = randomNumber(); + Object in = n; + Number out = n; + if (coerce && randomBoolean()) { + in = in.toString(); + } + return Tuple.tuple(in, out); + } + + private void mapping(XContentBuilder b) throws IOException { + minimalMapping(b); + if (coerce) { + b.field("coerce", true); + } + if (nullValue != null) { + b.field("null_value", nullValue); + } + } + + @Override + public List invalidExample() throws IOException { + return List.of( + new SyntheticSourceInvalidExample( + matchesPattern("field \\[field] of type \\[.+] doesn't support synthetic source because it doesn't have doc values"), + b -> { + minimalMapping(b); + b.field("doc_values", false); + } + ), + new SyntheticSourceInvalidExample( + matchesPattern("field \\[field] of type \\[.+] doesn't support synthetic source because it ignores malformed numbers"), + b -> { + minimalMapping(b); + b.field("ignore_malformed", true); + } + ) + ); + } + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/ShortFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/ShortFieldMapperTests.java index 179c1bd850b08..b78cdbb8f2bfb 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/ShortFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/ShortFieldMapperTests.java @@ -11,6 +11,7 @@ import org.elasticsearch.index.mapper.NumberFieldMapper.NumberType; import org.elasticsearch.index.mapper.NumberFieldTypeTests.OutOfRangeSpec; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.List; @@ -47,4 +48,9 @@ protected Number randomNumber() { } return randomDoubleBetween(Short.MIN_VALUE, Short.MAX_VALUE, true); } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/SourceFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/SourceFieldMapperTests.java index 4b45124733ed2..a1204e14d57c9 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/SourceFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/SourceFieldMapperTests.java @@ -44,6 +44,7 @@ protected void registerParameters(ParameterChecker checker) throws IOException { ); checker.registerConflictCheck("includes", b -> b.array("includes", "foo*")); checker.registerConflictCheck("excludes", b -> b.array("excludes", "foo*")); + checker.registerConflictCheck("synthetic", b -> b.field("synthetic", true)); } public void testNoFormat() throws Exception { diff --git a/server/src/test/java/org/elasticsearch/index/mapper/SourceLoaderTests.java b/server/src/test/java/org/elasticsearch/index/mapper/SourceLoaderTests.java new file mode 100644 index 0000000000000..d3660920d8a78 --- /dev/null +++ b/server/src/test/java/org/elasticsearch/index/mapper/SourceLoaderTests.java @@ -0,0 +1,77 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License + * 2.0 and the Server Side Public License, v 1; you may not use this file except + * in compliance with, at your election, the Elastic License 2.0 or the Server + * Side Public License, v 1. + */ + +package org.elasticsearch.index.mapper; + +import java.io.IOException; + +import static org.hamcrest.Matchers.equalTo; + +public class SourceLoaderTests extends MapperServiceTestCase { + public void testEmptyObject() throws IOException { + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("o").field("type", "object").endObject(); + b.startObject("kwd").field("type", "keyword").endObject(); + })); + assertThat(syntheticSource(mapper, b -> b.field("kwd", "foo")), equalTo(""" + {"kwd":"foo"}""")); + } + + public void testUnsupported() throws IOException { + Exception e = expectThrows( + IllegalArgumentException.class, + () -> createDocumentMapper(syntheticSourceMapping(b -> b.startObject("txt").field("type", "text").endObject())) + ); + assertThat( + e.getMessage(), + equalTo( + "field [txt] of type [text] doesn't support synthetic source unless" + + " it has a sub-field of type [keyword] with doc values enabled and without ignore_above or a normalizer" + ) + ); + } + + public void testDotsInFieldName() throws IOException { + DocumentMapper mapper = createDocumentMapper( + syntheticSourceMapping(b -> { b.startObject("foo.bar.baz").field("type", "keyword").endObject(); }) + ); + assertThat(syntheticSource(mapper, b -> b.field("foo.bar.baz", "aaa")), equalTo(""" + {"foo":{"bar":{"baz":"aaa"}}}""")); + } + + public void testSorted() throws IOException { + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("foo").field("type", "keyword").endObject(); + b.startObject("bar").field("type", "keyword").endObject(); + b.startObject("baz").field("type", "keyword").endObject(); + })); + assertThat( + syntheticSource(mapper, b -> b.field("foo", "over the lazy dog").field("bar", "the quick").field("baz", "brown fox jumped")), + equalTo(""" + {"bar":"the quick","baz":"brown fox jumped","foo":"over the lazy dog"}""") + ); + } + + public void testArraysPushedToLeaves() throws IOException { + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("o").startObject("properties"); + b.startObject("foo").field("type", "keyword").endObject(); + b.startObject("bar").field("type", "keyword").endObject(); + b.endObject().endObject(); + })); + assertThat(syntheticSource(mapper, b -> { + b.startArray("o"); + b.startObject().field("foo", "a").endObject(); + b.startObject().field("bar", "b").endObject(); + b.startObject().field("bar", "c").field("foo", "d").endObject(); + b.startObject().startArray("bar").value("e").value("f").endArray().endObject(); + b.endArray(); + }), equalTo(""" + {"o":{"bar":["b","c","e","f"],"foo":["a","d"]}}""")); + } +} diff --git a/server/src/test/java/org/elasticsearch/index/mapper/TextFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/TextFieldMapperTests.java index 55885bd80a2c9..6a5305b743eaf 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/TextFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/TextFieldMapperTests.java @@ -16,6 +16,7 @@ import org.apache.lucene.analysis.en.EnglishAnalyzer; import org.apache.lucene.analysis.standard.StandardAnalyzer; import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; +import org.apache.lucene.index.DirectoryReader; import org.apache.lucene.index.DocValuesType; import org.apache.lucene.index.IndexOptions; import org.apache.lucene.index.IndexableField; @@ -50,6 +51,7 @@ import org.elasticsearch.index.analysis.CharFilterFactory; import org.elasticsearch.index.analysis.CustomAnalyzer; import org.elasticsearch.index.analysis.IndexAnalyzers; +import org.elasticsearch.index.analysis.LowercaseNormalizer; import org.elasticsearch.index.analysis.NamedAnalyzer; import org.elasticsearch.index.analysis.StandardTokenizerFactory; import org.elasticsearch.index.analysis.TokenFilterFactory; @@ -62,11 +64,14 @@ import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; +import org.hamcrest.Matcher; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; import static org.hamcrest.Matchers.containsString; @@ -204,7 +209,7 @@ public TokenStream create(TokenStream tokenStream) { ); return new IndexAnalyzers( Map.of("default", dflt, "standard", standard, "keyword", keyword, "whitespace", whitespace, "my_stop_analyzer", stop), - Map.of(), + Map.of("lowercase", new NamedAnalyzer("lowercase", AnalyzerScope.INDEX, new LowercaseNormalizer())), Map.of() ); } @@ -1087,4 +1092,78 @@ protected Object generateRandomInputValue(MappedFieldType ft) { protected void randomFetchTestFieldConfig(XContentBuilder b) throws IOException { assumeFalse("We don't have a way to assert things here", true); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + SyntheticSourceExample delegate = new KeywordFieldMapperTests.KeywordSyntheticSourceSupport().example(); + return new SyntheticSourceSupport() { + @Override + public SyntheticSourceExample example() throws IOException { + return new SyntheticSourceExample(delegate.inputValue(), delegate.result(), b -> { + b.field("type", "text"); + b.startObject("fields"); + { + b.startObject(randomAlphaOfLength(4)); + delegate.mapping().accept(b); + b.endObject(); + } + b.endObject(); + }); + } + + @Override + public List invalidExample() throws IOException { + Matcher err = equalTo( + "field [field] of type [text] doesn't support synthetic source " + + "unless it has a sub-field of type [keyword] with doc values enabled and without ignore_above or a normalizer" + ); + return List.of( + new SyntheticSourceInvalidExample(err, TextFieldMapperTests.this::minimalMapping), + new SyntheticSourceInvalidExample(err, b -> { + b.field("type", "text"); + b.startObject("fields"); + { + b.startObject("l"); + b.field("type", "long"); + b.endObject(); + } + b.endObject(); + }), + new SyntheticSourceInvalidExample(err, b -> { + b.field("type", "text"); + b.startObject("fields"); + { + b.startObject("kwd"); + b.field("type", "keyword"); + b.field("ignore_above", 10); + b.endObject(); + } + b.endObject(); + }), + new SyntheticSourceInvalidExample(err, b -> { + b.field("type", "text"); + b.startObject("fields"); + { + b.startObject("kwd"); + b.field("type", "keyword"); + b.field("normalizer", "lowercase"); + b.endObject(); + } + b.endObject(); + }) + ); + } + }; + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected void validateRoundTripReader(String syntheticSource, DirectoryReader reader, DirectoryReader roundTripReader) + throws IOException { + // Disabled because it currently fails + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/WholeNumberFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/WholeNumberFieldMapperTests.java index 52931478b93b7..7b201f59001e5 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/WholeNumberFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/WholeNumberFieldMapperTests.java @@ -100,4 +100,8 @@ protected void registerParameters(ParameterChecker checker) throws IOException { registerDimensionChecks(checker); } + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + return new NumberSyntheticSourceSupport(Number::longValue); + } } diff --git a/server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java b/server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java index 2dab44f85c3cc..d0b3617934b8e 100644 --- a/server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java +++ b/server/src/test/java/org/elasticsearch/index/mapper/flattened/FlattenedFieldMapperTests.java @@ -26,6 +26,7 @@ import org.elasticsearch.index.mapper.flattened.FlattenedFieldMapper.RootFlattenedFieldType; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentType; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Arrays; @@ -421,4 +422,14 @@ public void testDynamicTemplateAndDottedPaths() throws IOException { IndexableField[] keyed = doc.rootDoc().getFields("a.b.c._keyed"); assertEquals(new BytesRef("d\0value"), keyed[0].binaryValue()); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperServiceTestCase.java b/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperServiceTestCase.java index 9ab8670b7752d..7a34c80eec74d 100644 --- a/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperServiceTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperServiceTestCase.java @@ -10,6 +10,7 @@ import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.standard.StandardAnalyzer; +import org.apache.lucene.index.DirectoryReader; import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.IndexWriterConfig; import org.apache.lucene.search.IndexSearcher; @@ -61,6 +62,7 @@ import org.elasticsearch.search.sort.SortAndFormats; import org.elasticsearch.search.sort.SortBuilder; import org.elasticsearch.test.ESTestCase; +import org.elasticsearch.test.FieldMaskingReader; import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; @@ -81,6 +83,7 @@ import static java.util.Collections.emptyList; import static java.util.stream.Collectors.toList; +import static org.hamcrest.Matchers.equalTo; import static org.mockito.Mockito.mock; /** @@ -644,4 +647,62 @@ protected BiFunction, IndexFieldData> return (mft, lookupSource) -> mft.fielddataBuilder("test", lookupSource) .build(new IndexFieldDataCache.None(), new NoneCircuitBreakerService()); } + + protected final String syntheticSource(DocumentMapper mapper, CheckedConsumer build) throws IOException { + try (Directory directory = newDirectory()) { + RandomIndexWriter iw = new RandomIndexWriter(random(), directory); + iw.addDocument(mapper.parse(source(build)).rootDoc()); + iw.close(); + try (DirectoryReader reader = DirectoryReader.open(directory)) { + SourceLoader loader = mapper.sourceMapper().newSourceLoader(mapper.mapping().getRoot()); + String syntheticSource = loader.leaf(getOnlyLeafReader(reader)).source(null, 0).utf8ToString(); + roundTripSyntheticSource(mapper, syntheticSource, reader); + return syntheticSource; + } + } + } + + /* + * Use the synthetic source to build a *second* index and verify + * that the synthetic source it produces is the same. And *then* + * verify that the index's contents are exactly the same. Except + * for _recovery_source - that won't be the same because it's a + * precise copy of the bits in _source. And we *know* that frequently + * the synthetic source won't be the same as the original source. + * That's the point, really. It'll just be "close enough" for + * round tripping. + */ + private void roundTripSyntheticSource(DocumentMapper mapper, String syntheticSource, DirectoryReader reader) throws IOException { + try (Directory roundTripDirectory = newDirectory()) { + RandomIndexWriter roundTripIw = new RandomIndexWriter(random(), roundTripDirectory); + roundTripIw.addDocument( + mapper.parse(new SourceToParse("1", new BytesArray(syntheticSource), XContentType.JSON, null, Map.of())).rootDoc() + ); + roundTripIw.close(); + try (DirectoryReader roundTripReader = DirectoryReader.open(roundTripDirectory)) { + SourceLoader loader = mapper.sourceMapper().newSourceLoader(mapper.mapping().getRoot()); + String roundTripSyntheticSource = loader.leaf(getOnlyLeafReader(roundTripReader)).source(null, 0).utf8ToString(); + assertThat(roundTripSyntheticSource, equalTo(syntheticSource)); + validateRoundTripReader(syntheticSource, reader, roundTripReader); + } + } + } + + protected void validateRoundTripReader(String syntheticSource, DirectoryReader reader, DirectoryReader roundTripReader) + throws IOException { + assertReaderEquals( + "round trip " + syntheticSource, + new FieldMaskingReader(SourceFieldMapper.RECOVERY_SOURCE_NAME, reader), + new FieldMaskingReader(SourceFieldMapper.RECOVERY_SOURCE_NAME, roundTripReader) + ); + } + + protected final XContentBuilder syntheticSourceMapping(CheckedConsumer buildFields) throws IOException { + return topMapping(b -> { + b.startObject("_source").field("synthetic", true).endObject(); + b.startObject("properties"); + buildFields.accept(b); + b.endObject(); + }); + } } diff --git a/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperTestCase.java b/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperTestCase.java index aa03a68eb8a01..3ea262bb9fbf8 100644 --- a/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/index/mapper/MapperTestCase.java @@ -30,6 +30,9 @@ import org.elasticsearch.index.fielddata.IndexFieldDataCache; import org.elasticsearch.index.query.SearchExecutionContext; import org.elasticsearch.indices.breaker.NoneCircuitBreakerService; +import org.elasticsearch.script.Script; +import org.elasticsearch.script.ScriptContext; +import org.elasticsearch.script.ScriptFactory; import org.elasticsearch.script.field.DocValuesScriptFieldFactory; import org.elasticsearch.search.DocValueFormat; import org.elasticsearch.search.lookup.LeafStoredFieldsLookup; @@ -38,6 +41,7 @@ import org.elasticsearch.xcontent.ToXContent; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.json.JsonXContent; +import org.hamcrest.Matcher; import java.io.IOException; import java.util.ArrayList; @@ -45,6 +49,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.function.Consumer; import java.util.function.Function; @@ -56,6 +61,7 @@ import static org.hamcrest.Matchers.containsString; import static org.hamcrest.Matchers.equalTo; import static org.hamcrest.Matchers.instanceOf; +import static org.hamcrest.Matchers.matchesPattern; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @@ -763,4 +769,142 @@ protected String minimalIsInvalidRoutingPathErrorMessage(Mapper mapper) { + mapper.typeName() + "]."; } + + public record SyntheticSourceExample(Object inputValue, Object result, CheckedConsumer mapping) {} + + public record SyntheticSourceInvalidExample(Matcher error, CheckedConsumer mapping) {} + + public interface SyntheticSourceSupport { + /** + * Examples that should work when source is generated from doc values. + */ + SyntheticSourceExample example() throws IOException; + + /** + * Examples of mappings that should be rejected when source is configured to + * be loaded from doc values. + */ + List invalidExample() throws IOException; + } + + protected abstract SyntheticSourceSupport syntheticSourceSupport(); + + public final void testSyntheticSource() throws IOException { + SyntheticSourceExample syntheticSourceExample = syntheticSourceSupport().example(); + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("field"); + syntheticSourceExample.mapping().accept(b); + b.endObject(); + })); + String expected = Strings.toString( + JsonXContent.contentBuilder().startObject().field("field", syntheticSourceExample.result).endObject() + ); + assertThat(syntheticSource(mapper, b -> b.field("field", syntheticSourceExample.inputValue)), equalTo(expected)); + } + + public final void testNoSyntheticSourceForScript() throws IOException { + // Fetch the ingest script support to eagerly assumeFalse if the mapper doesn't support ingest scripts + ingestScriptSupport(); + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("field"); + minimalMapping(b); + b.field("script", randomBoolean() ? "empty" : "non-empty"); + b.endObject(); + })); + assertThat(syntheticSource(mapper, b -> {}), equalTo("{}")); + } + + public final void testSyntheticSourceInObject() throws IOException { + SyntheticSourceExample syntheticSourceExample = syntheticSourceSupport().example(); + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("obj").startObject("properties").startObject("field"); + syntheticSourceExample.mapping().accept(b); + b.endObject().endObject().endObject(); + })); + String expected = Strings.toString( + JsonXContent.contentBuilder() + .startObject() + .startObject("obj") + .field("field", syntheticSourceExample.result) + .endObject() + .endObject() + ); + assertThat( + syntheticSource(mapper, b -> b.startObject("obj").field("field", syntheticSourceExample.inputValue).endObject()), + equalTo(expected) + ); + } + + public final void testSyntheticEmptyList() throws IOException { + SyntheticSourceExample syntheticSourceExample = syntheticSourceSupport().example(); + DocumentMapper mapper = createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("field"); + syntheticSourceExample.mapping().accept(b); + b.endObject(); + })); + assertThat(syntheticSource(mapper, b -> b.startArray("field").endArray()), equalTo("{}")); + } + + public final void testSyntheticSourceInvalid() throws IOException { + List examples = new ArrayList<>(syntheticSourceSupport().invalidExample()); + examples.add( + new SyntheticSourceInvalidExample( + matchesPattern("field \\[field] of type \\[.+] doesn't support synthetic source because it declares copy_to"), + b -> { + syntheticSourceSupport().example().mapping().accept(b); + b.field("copy_to", "bar"); + } + ) + ); + for (SyntheticSourceInvalidExample example : examples) { + Exception e = expectThrows( + IllegalArgumentException.class, + example.toString(), + () -> createDocumentMapper(syntheticSourceMapping(b -> { + b.startObject("field"); + example.mapping.accept(b); + b.endObject(); + })) + ); + assertThat(e.getMessage(), example.error); + } + } + + @Override + protected final T compileScript(Script script, ScriptContext context) { + return ingestScriptSupport().compileScript(script, context); + } + + protected abstract IngestScriptSupport ingestScriptSupport(); + + protected abstract class IngestScriptSupport { + private T compileScript(Script script, ScriptContext context) { + switch (script.getIdOrCode()) { + case "empty": + return context.factoryClazz.cast(emptyFieldScript()); + case "non-empty": + return context.factoryClazz.cast(nonEmptyFieldScript()); + default: + return compileOtherScript(script, context); + } + } + + protected T compileOtherScript(Script script, ScriptContext context) { + throw new UnsupportedOperationException("Unknown script " + script.getIdOrCode()); + } + + /** + * Create a script that can be run to produce no values for this + * field or return {@link Optional#empty()} to signal that this + * field doesn't support fields scripts. + */ + abstract ScriptFactory emptyFieldScript(); + + /** + * Create a script that can be run to produce some value value for this + * field or return {@link Optional#empty()} to signal that this + * field doesn't support fields scripts. + */ + abstract ScriptFactory nonEmptyFieldScript(); + } } diff --git a/test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java b/test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java index 1b5d2cddb3da9..314c43a37775f 100644 --- a/test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java +++ b/test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java @@ -1645,6 +1645,16 @@ protected Map getIndexSettingsAsMap(String index) throws IOExcep return (Map) ((Map) indexSettings.get(index)).get("settings"); } + protected static Map getIndexMapping(String index) throws IOException { + return entityAsMap(client().performRequest(new Request("GET", "/" + index + "/_mapping"))); + } + + @SuppressWarnings("unchecked") + protected Map getIndexMappingAsMap(String index) throws IOException { + Map indexSettings = getIndexMapping(index); + return (Map) ((Map) indexSettings.get(index)).get("mappings"); + } + protected static boolean indexExists(String index) throws IOException { Response response = client().performRequest(new Request("HEAD", "/" + index)); return RestStatus.OK.getStatus() == response.getStatusLine().getStatusCode(); diff --git a/x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapperTests.java b/x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapperTests.java index 173083100ce97..5e6f86f7411c6 100644 --- a/x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapperTests.java +++ b/x-pack/plugin/analytics/src/test/java/org/elasticsearch/xpack/analytics/mapper/HistogramFieldMapperTests.java @@ -16,6 +16,7 @@ import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xpack.analytics.AnalyticsPlugin; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -345,4 +346,13 @@ public void testMetricType() throws IOException { } } + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/ccr/qa/multi-cluster/src/test/java/org/elasticsearch/xpack/ccr/FollowIndexIT.java b/x-pack/plugin/ccr/qa/multi-cluster/src/test/java/org/elasticsearch/xpack/ccr/FollowIndexIT.java index ba0706656b6e4..8d2ca6a21fd55 100644 --- a/x-pack/plugin/ccr/qa/multi-cluster/src/test/java/org/elasticsearch/xpack/ccr/FollowIndexIT.java +++ b/x-pack/plugin/ccr/qa/multi-cluster/src/test/java/org/elasticsearch/xpack/ccr/FollowIndexIT.java @@ -381,6 +381,64 @@ public void testFollowStandardIndexCanNotOverrideMode() throws Exception { ); } + public void testSyntheticSource() throws Exception { + final int numDocs = 128; + final String leaderIndexName = "synthetic_leader"; + long basetime = DateFieldMapper.DEFAULT_DATE_TIME_FORMATTER.parseMillis("2021-04-28T18:35:24.467Z"); + if ("leader".equals(targetCluster)) { + logger.info("Running against leader cluster"); + createIndex(adminClient(), leaderIndexName, Settings.EMPTY, """ + "_source": {"synthetic": true}, + "properties": {"kwd": {"type": "keyword"}}}""", null); + for (int i = 0; i < numDocs; i++) { + logger.info("Indexing doc [{}]", i); + index(client(), leaderIndexName, null, "kwd", "foo", "i", i); + } + refresh(adminClient(), leaderIndexName); + verifyDocuments(client(), leaderIndexName, numDocs); + } else if ("follow".equals(targetCluster)) { + logger.info("Running against follow cluster"); + final String followIndexName = "synthetic_follower"; + final boolean overrideNumberOfReplicas = randomBoolean(); + if (overrideNumberOfReplicas) { + followIndex( + client(), + "leader_cluster", + leaderIndexName, + followIndexName, + Settings.builder().put("index.number_of_replicas", 0).build() + ); + } else { + followIndex(leaderIndexName, followIndexName); + } + assertBusy(() -> { + verifyDocuments(client(), followIndexName, numDocs); + assertMap(getIndexMappingAsMap(followIndexName), matchesMap().extraOk().entry("_source", Map.of("synthetic", true))); + if (overrideNumberOfReplicas) { + assertMap(getIndexSettingsAsMap(followIndexName), matchesMap().extraOk().entry("index.number_of_replicas", "0")); + } else { + assertMap(getIndexSettingsAsMap(followIndexName), matchesMap().extraOk().entry("index.number_of_replicas", "1")); + } + }); + // unfollow and then follow and then index a few docs in leader index: + pauseFollow(followIndexName); + resumeFollow(followIndexName); + try (RestClient leaderClient = buildLeaderClient()) { + index(leaderClient, leaderIndexName, null, "kwd", "foo", "i", -1); + index(leaderClient, leaderIndexName, null, "kwd", "foo", "i", -2); + index(leaderClient, leaderIndexName, null, "kwd", "foo", "i", -3); + } + assertBusy(() -> verifyDocuments(client(), followIndexName, numDocs + 3)); + assertBusy(() -> verifyCcrMonitoring(leaderIndexName, followIndexName), 30, TimeUnit.SECONDS); + + pauseFollow(followIndexName); + closeIndex(followIndexName); + assertOK(client().performRequest(new Request("POST", "/" + followIndexName + "/_ccr/unfollow"))); + Exception e = expectThrows(ResponseException.class, () -> resumeFollow(followIndexName)); + assertThat(e.getMessage(), containsString("follow index [" + followIndexName + "] does not have ccr metadata")); + } + } + @Override protected Settings restClientSettings() { String token = basicAuthHeaderValue("admin", new SecureString("admin-password".toCharArray())); diff --git a/x-pack/plugin/mapper-aggregate-metric/src/test/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapperTests.java b/x-pack/plugin/mapper-aggregate-metric/src/test/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapperTests.java index 8d35fe260c879..b51f522f2dcc3 100644 --- a/x-pack/plugin/mapper-aggregate-metric/src/test/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapperTests.java +++ b/x-pack/plugin/mapper-aggregate-metric/src/test/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateDoubleMetricFieldMapperTests.java @@ -21,6 +21,7 @@ import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xpack.aggregatemetric.AggregateMetricMapperPlugin; import org.hamcrest.Matchers; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -586,4 +587,14 @@ public void testMetricType() throws IOException { ); } } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/mapper-constant-keyword/src/internalClusterTest/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapperTests.java b/x-pack/plugin/mapper-constant-keyword/src/internalClusterTest/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapperTests.java index 0890b033fa4c5..e9077f3cb8a97 100644 --- a/x-pack/plugin/mapper-constant-keyword/src/internalClusterTest/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapperTests.java +++ b/x-pack/plugin/mapper-constant-keyword/src/internalClusterTest/java/org/elasticsearch/xpack/constantkeyword/mapper/ConstantKeywordFieldMapperTests.java @@ -24,6 +24,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xpack.constantkeyword.ConstantKeywordMapperPlugin; import org.elasticsearch.xpack.constantkeyword.mapper.ConstantKeywordFieldMapper.ConstantKeywordFieldType; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -203,4 +204,14 @@ protected void randomFetchTestFieldConfig(XContentBuilder b) throws IOException protected boolean allowsNullValues() { return false; // null is an error for constant keyword } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/mapper-unsigned-long/src/test/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapperTests.java b/x-pack/plugin/mapper-unsigned-long/src/test/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapperTests.java index 04aef6e4c2f8d..00f7ba6c72145 100644 --- a/x-pack/plugin/mapper-unsigned-long/src/test/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapperTests.java +++ b/x-pack/plugin/mapper-unsigned-long/src/test/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapperTests.java @@ -19,6 +19,7 @@ import org.elasticsearch.index.termvectors.TermVectorsService; import org.elasticsearch.plugins.Plugin; import org.elasticsearch.xcontent.XContentBuilder; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.math.BigInteger; @@ -375,4 +376,14 @@ private Number randomNumericValue() { return big.add(randomBoolean() ? BigInteger.ONE : BigInteger.ZERO); } } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/mapper-version/src/test/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapperTests.java b/x-pack/plugin/mapper-version/src/test/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapperTests.java index 56f1b74fc4480..5e353d7fb47ad 100644 --- a/x-pack/plugin/mapper-version/src/test/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapperTests.java +++ b/x-pack/plugin/mapper-version/src/test/java/org/elasticsearch/xpack/versionfield/VersionStringFieldMapperTests.java @@ -23,6 +23,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentType; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -159,4 +160,14 @@ private String randomPrerelease() { protected boolean dedupAfterFetch() { return true; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapperTests.java b/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapperTests.java index fe1f82fde5ea3..7085da9b8b1b7 100644 --- a/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapperTests.java +++ b/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/GeoShapeWithDocValuesFieldMapperTests.java @@ -26,6 +26,7 @@ import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xcontent.XContentType; import org.elasticsearch.xpack.spatial.LocalStateSpatialPlugin; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collection; @@ -426,4 +427,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapperTests.java b/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapperTests.java index 147a56b67eccd..e6265064e63d2 100644 --- a/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapperTests.java +++ b/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/PointFieldMapperTests.java @@ -18,6 +18,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xcontent.XContentFactory; import org.elasticsearch.xpack.spatial.common.CartesianPoint; +import org.junit.AssumptionViolatedException; import java.io.IOException; @@ -368,4 +369,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapperTests.java b/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapperTests.java index aa28af87dbc1c..d2a96069d8c80 100644 --- a/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapperTests.java +++ b/x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapperTests.java @@ -18,6 +18,7 @@ import org.elasticsearch.index.mapper.ParsedDocument; import org.elasticsearch.index.mapper.SourceToParse; import org.elasticsearch.xcontent.ToXContent; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.util.Collections; @@ -277,4 +278,14 @@ protected Object generateRandomInputValue(MappedFieldType ft) { assumeFalse("Test implemented in a follow up", true); return null; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/vectors/src/test/java/org/elasticsearch/xpack/vectors/mapper/DenseVectorFieldMapperTests.java b/x-pack/plugin/vectors/src/test/java/org/elasticsearch/xpack/vectors/mapper/DenseVectorFieldMapperTests.java index 3f5819c4c080c..4ef0c39d87d3b 100644 --- a/x-pack/plugin/vectors/src/test/java/org/elasticsearch/xpack/vectors/mapper/DenseVectorFieldMapperTests.java +++ b/x-pack/plugin/vectors/src/test/java/org/elasticsearch/xpack/vectors/mapper/DenseVectorFieldMapperTests.java @@ -34,6 +34,7 @@ import org.elasticsearch.xpack.vectors.DenseVectorPlugin; import org.elasticsearch.xpack.vectors.mapper.DenseVectorFieldMapper.DenseVectorFieldType; import org.elasticsearch.xpack.vectors.mapper.DenseVectorFieldMapper.VectorSimilarity; +import org.junit.AssumptionViolatedException; import java.io.IOException; import java.nio.ByteBuffer; @@ -476,4 +477,14 @@ public void testKnnVectorsFormat() throws IOException { + ")"; assertEquals(expectedString, knnVectorsFormat.toString()); } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/plugin/wildcard/src/test/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapperTests.java b/x-pack/plugin/wildcard/src/test/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapperTests.java index 9d7c6c757cb11..def4b9fb7abd0 100644 --- a/x-pack/plugin/wildcard/src/test/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapperTests.java +++ b/x-pack/plugin/wildcard/src/test/java/org/elasticsearch/xpack/wildcard/mapper/WildcardFieldMapperTests.java @@ -69,6 +69,7 @@ import org.elasticsearch.xcontent.XContentBuilder; import org.elasticsearch.xpack.wildcard.Wildcard; import org.elasticsearch.xpack.wildcard.mapper.WildcardFieldMapper.Builder; +import org.junit.AssumptionViolatedException; import org.junit.Before; import org.mockito.Mockito; @@ -1204,4 +1205,14 @@ protected String generateRandomInputValue(MappedFieldType ft) { protected boolean dedupAfterFetch() { return true; } + + @Override + protected SyntheticSourceSupport syntheticSourceSupport() { + throw new AssumptionViolatedException("not supported"); + } + + @Override + protected IngestScriptSupport ingestScriptSupport() { + throw new AssumptionViolatedException("not supported"); + } } diff --git a/x-pack/qa/runtime-fields/build.gradle b/x-pack/qa/runtime-fields/build.gradle index a01b3765745ab..7bdaf713a1f19 100644 --- a/x-pack/qa/runtime-fields/build.gradle +++ b/x-pack/qa/runtime-fields/build.gradle @@ -104,6 +104,8 @@ subprojects { // we need a @timestamp field to be defined in index mapping 'search/380_sort_segments_on_timestamp/*', 'field_caps/40_time_series/*', + // Synthetic source needs doc values that runtime fields tests disable + 'search/400_synthetic_source/*', /////// NOT SUPPORTED /////// ].join(',') }