opensearch-project · kolchfa-aws · Sep 22, 2023 · Aug 28, 2023 · Aug 29, 2023 · Sep 6, 2023
@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Boolean queries
+title: Boolean
 parent: Compound queries
 grand_parent: Query DSL
 nav_order: 10

@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Boosting queries
+title: Boosting
 parent: Compound queries
 grand_parent: Query DSL
 nav_order: 30

@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Constant score queries
+title: Constant score
 parent: Compound queries
 grand_parent: Query DSL
 nav_order: 40

@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Disjunction max queries
+title: Disjunction max
 parent: Compound queries
 grand_parent: Query DSL
 nav_order: 50

@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Function score queries
+title: Function score
 parent: Compound queries
 grand_parent: Query DSL
 nav_order: 60

@@ -0,0 +1,55 @@
+---
+layout: default
+title: Hybrid
+parent: Compound queries
+grand_parent: Query DSL
+nav_order: 70
+---
+
+# Hybrid query
+
+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https:/opensearch-project/neural-search/issues/244). 
+{: .warning}
+
+Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.
-Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.
+Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries and calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.
-Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.
+Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries and calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.
+
+## Example
+
+The following example request combines a score from a regular `match` query clause with a score from a `neural` query clause. It uses a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/) with a [normalization processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/script-processor/), which specifies the techniques to normalize and combine query clause relevance scores:
+
+```json
+POST flicker-index/_search?search_pipeline=normalizationPipeline
+{
+ "query": {
+ "hybrid": {
+ "queries": [
+ {
+ "neural": {
+ "passage_embedding": {
+ "query_text": "Girl with Brown Hair",
+ "model_id": "ABCBMODELID",
+ "k": 20
+ }
+ }
+ },
+ {
+ "match": {
+ "passage_text": "Girl Brown hair"
+ }
+ }
+ ]
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+To learn more about the normalization processor, see [Normalization processor]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/normalization-processor/).
+
+## Parameters
+
+The following table lists all top-level parameters supported by `hybrid` queries.
+
+Parameter | Description
+:--- | :---
+`queries` | An array of one or more query clauses that are used to match documents. A document must match at least one query clause to be returned in the results. The documents' relevance scores from all query clauses are combined into one score by applying a [search pipeline]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/). The maximum number of query clauses is 5. Required.
@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Filter query processor
+title: Filter query
 nav_order: 10
 has_children: false
 parent: Search processors

@@ -14,9 +14,10 @@ You can use _search pipelines_ to build new or reuse existing result rerankers,
 
 The following is a list of search pipeline terminology:
 
-* _Search request processor_: A component that takes a search request (the query and the metadata passed in the request), performs an operation with or on the search request, and returns a search request.
-* _Search response processor_: A component that takes a search response and search request (the query, results, and metadata passed in the request), performs an operation with or on the search response, and returns a search response.
-* _Processor_: Either a search request processor or a search response processor.
+* [_Search request processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-request-processors): A component that takes a search request (the query and the metadata passed in the request), performs an operation with or on the search request, and returns a search request.
+* [_Search response processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-response-processors): A component that takes a search response and search request (the query, results, and metadata passed in the request), performs an operation with or on the search response, and returns a search response.
+* [_Search phase results processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-phase-results-processors): A component that runs between search phases at the coordinating node level. A search phase results processor takes the results retrieved from one search phase and transforms them before passing them to the next search phase.
-* [_Search phase results processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-phase-results-processors): A component that runs between search phases at the coordinating node level. A search phase results processor takes the results retrieved from one search phase and transforms them before passing them to the next search phase.
+* [_Search phase results processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-phase-results-processors): A component that runs between search phases at the coordinating node level. A search phase results processor transforms the results retrieved from one search phase before passing them to the next search phase.
-* [_Search phase results processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-phase-results-processors): A component that runs between search phases at the coordinating node level. A search phase results processor takes the results retrieved from one search phase and transforms them before passing them to the next search phase.
+* [_Search phase results processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors#search-phase-results-processors): A component that runs between search phases at the coordinating node level. A search phase results processor transforms the results retrieved from one search phase before passing them to the next search phase.
+* [_Processor_]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/search-processors/): Either a search request processor or a search response processor.
 * _Search pipeline_: An ordered list of processors that is integrated into OpenSearch. The pipeline intercepts a query, performs processing on the query, sends it to OpenSearch, intercepts the results, performs processing on the results, and returns them to the calling application, as shown in the following diagram. 
 
 ![Search processor diagram]({{site.url}}{{site.baseurl}}/images/search-pipelines.png)

@@ -0,0 +1,103 @@
+---
+layout: default
+title: Normalization
+nav_order: 15
+has_children: false
+parent: Search processors
+grand_parent: Search pipelines
+---
+
+# Normalization processor
+
+The `normalization_processor` is a search phase results processor that runs between the query and fetch phases of search. It intercepts the query phase results and then normalizes and combines the document scores from different query clauses before passing the documents to the fetch phase.
-The `normalization_processor` is a search phase results processor that runs between the query and fetch phases of search. It intercepts the query phase results and then normalizes and combines the document scores from different query clauses before passing the documents to the fetch phase.
+The `normalization_processor` is a search phase results processor that runs between the query and fetch search phases. It intercepts the query phase results and then normalizes and combines the document scores from different query clauses before passing the documents to the fetch phase.
-The `normalization_processor` is a search phase results processor that runs between the query and fetch phases of search. It intercepts the query phase results and then normalizes and combines the document scores from different query clauses before passing the documents to the fetch phase.
+The `normalization_processor` is a search phase results processor that runs between the query and fetch search phases. It intercepts the query phase results and then normalizes and combines the document scores from different query clauses before passing the documents to the fetch phase.
+
+## Score normalization and combination
+
+Many applications require both keyword matching and semantic understanding. For example, BM25 accurately provides relevant search results for a query containing keywords, and neural networks perform well when a query requires natural language understanding. Thus, you might want to combine BM25 search results with the results of k-NN or neural search. However, BM25 and k-NN search use different scales to calculate relevance scores for the matching documents. Before combining the scores from multiple queries, it is necessary to normalize those scores so they are on the same scale. For further reading about score normalization and combination, including benchmarks and discussion of various techniques, see this [semantic search blog](https://opensearch.org/blog/semantic-science-benchmarks/).
+
+## Query then fetch
+
+OpenSearch supports two search types: `query_then_fetch` and `dfs_query_then_fetch`. The following diagram outlines the query then fetch process that includes a normalization processor.
-OpenSearch supports two search types: `query_then_fetch` and `dfs_query_then_fetch`. The following diagram outlines the query then fetch process that includes a normalization processor.
+OpenSearch supports two search types: `query_then_fetch` and `dfs_query_then_fetch`. The following diagram outlines the query-then-fetch process, which includes a normalization processor.
-OpenSearch supports two search types: `query_then_fetch` and `dfs_query_then_fetch`. The following diagram outlines the query then fetch process that includes a normalization processor.
+OpenSearch supports two search types: `query_then_fetch` and `dfs_query_then_fetch`. The following diagram outlines the query-then-fetch process, which includes a normalization processor.
+
+![Normalization processor flow diagram]({{site.url}}{{site.baseurl}}/images/normalization-processor.png)
+
+When you send a search request to a node, this node becomes a _coordinating node_. During the first phase of search, the _query phase_, the coordinating node routes the search request to all shards in the index, including primary and replica shards. Each shard then runs the search query locally and returns metadata about the matching documents, which includes their doc IDs and relevance scores. The `normalization_processor` then normalizes and combines scores from different query clauses. The coordinating node merges and sorts the local result lists, compiling a global list of top documents that match the query. After that, search enters a _fetch phase_, in which the coordinating node requests the documents in the global list from the shards where they reside. Each shard returns the documents' `_source` to the coordinating node. Finally, the coordinating node sends a search response containing the results back to you.
-When you send a search request to a node, this node becomes a _coordinating node_. During the first phase of search, the _query phase_, the coordinating node routes the search request to all shards in the index, including primary and replica shards. Each shard then runs the search query locally and returns metadata about the matching documents, which includes their doc IDs and relevance scores. The `normalization_processor` then normalizes and combines scores from different query clauses. The coordinating node merges and sorts the local result lists, compiling a global list of top documents that match the query. After that, search enters a _fetch phase_, in which the coordinating node requests the documents in the global list from the shards where they reside. Each shard returns the documents' `_source` to the coordinating node. Finally, the coordinating node sends a search response containing the results back to you.
+When you send a search request to a node, it becomes a _coordinating node_. During the first phase of search, the _query phase_, the coordinating node routes the search request to all shards in the index, including primary and replica shards. Each shard then runs the search query locally and returns metadata about the matching documents, which includes their document IDs and relevance scores. The `normalization_processor` then normalizes and combines scores from different query clauses. The coordinating node merges and sorts the local result lists, compiling a global list of top documents that match the query. After that is the _fetch phase_, in which the coordinating node requests the documents in the global list from the shards where they reside. Each shard returns the documents' `_source` to the coordinating node. Finally, the coordinating node sends a search response containing the results.
-When you send a search request to a node, this node becomes a _coordinating node_. During the first phase of search, the _query phase_, the coordinating node routes the search request to all shards in the index, including primary and replica shards. Each shard then runs the search query locally and returns metadata about the matching documents, which includes their doc IDs and relevance scores. The `normalization_processor` then normalizes and combines scores from different query clauses. The coordinating node merges and sorts the local result lists, compiling a global list of top documents that match the query. After that, search enters a _fetch phase_, in which the coordinating node requests the documents in the global list from the shards where they reside. Each shard returns the documents' `_source` to the coordinating node. Finally, the coordinating node sends a search response containing the results back to you.
+When you send a search request to a node, it becomes a _coordinating node_. During the first phase of search, the _query phase_, the coordinating node routes the search request to all shards in the index, including primary and replica shards. Each shard then runs the search query locally and returns metadata about the matching documents, which includes their document IDs and relevance scores. The `normalization_processor` then normalizes and combines scores from different query clauses. The coordinating node merges and sorts the local result lists, compiling a global list of top documents that match the query. After that is the _fetch phase_, in which the coordinating node requests the documents in the global list from the shards where they reside. Each shard returns the documents' `_source` to the coordinating node. Finally, the coordinating node sends a search response containing the results.
+
+## Request fields
+
+The following table lists all available request fields.
+
+Field | Data type | Description
+:--- | :--- | :---
+`normalization.technique` | String | The technique for normalizing scores. Valid values are `min_max`, `L2`. Optional. Default is `min_max`.
+`combination.technique` | String | The technique for combining scores. Valid values are `harmonic_mean`, `arithmetic_mean`, `geometric_mean`. Optional. Default is `arithmetic_mean`.
+`combination.parameters.weights` | Array of floating-point values | Specifies the weights to use for each query. Valid values are in the [0.0, 1.0] range and signify decimal percentages. The closer the weight is to 1.0, the more weight is given to a query. The number of values in the `weights` array must equal the number of queries. The sum of the values in the array must equal 1.0. Optional. If not provided, all queries are given equal weight.
+`tag` | String | The processor's identifier. Optional.
+`description` | String | A description of the processor. Optional.
+`ignore_failure` | Boolean | If `true`, OpenSearch [ignores a failure]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/#ignoring-processor-failures) of this processor and continues to run the remaining processors in the search pipeline. Optional. Default is `false`.
+
+## Example 
+
+The following example demonstrates using a search pipeline with a `normalization_processor`.
+
+### Creating a search pipeline 
+
+The following request creates a search pipeline with a `normalization_processor` that uses the `min_max` normalization technique and the `harmonic_mean` combination technique:
+
+```json
+PUT /_search/pipeline/my_pipeline
+{
+ "phase_results_processors" : [
+ {
+ "normalization-processor" : {
+ "normalization": { 
+ "technique": "min_max", 
+ },
+ "combination": { 
+ "technique" : "arithmetic_mean", 
+ "parameters" : { 
+ "weights" : [0.4, 0.7] 
+ }
+ }
+ }
+ }
+ ]
+}
+```
+{% include copy-curl.html %}
+
+### Using a search pipeline
+
+Provide the query clauses that you want to combine in a `hybrid` query and apply the search pipeline created in the previous section so the scores are combined using the chosen techniques:
-Provide the query clauses that you want to combine in a `hybrid` query and apply the search pipeline created in the previous section so the scores are combined using the chosen techniques:
+Provide the query clauses that you want to combine in a `hybrid` query and apply the search pipeline created in the previous section so that the scores are combined using the chosen techniques:
-Provide the query clauses that you want to combine in a `hybrid` query and apply the search pipeline created in the previous section so the scores are combined using the chosen techniques:
+Provide the query clauses that you want to combine in a `hybrid` query and apply the search pipeline created in the previous section so that the scores are combined using the chosen techniques:
+
+```json
+POST flicker-index/_search?search_pipeline=normalizationPipeline
+{
+ "query": {
+ "hybrid": {
+ "queries": [
+ {
+ "neural": {
+ "passage_embedding": {
+ "query_text": "Girl with Brown Hair",
+ "model_id": "ABCBMODELID",
+ "k": 20
+ }
+ }
+ },
+ {
+ "match": {
+ "passage_text": "Girl Brown hair"
+ }
+ }
+ ]
+ }
+ }
+}
+```
+{% include copy-curl.html %}
+
+For more information, see [Hybrid query]({{site.url}}{{site.baseurl}}/query-dsl/compound/hybrid/).
+
+The `normalization_processor` does not produce consistent results for a cluster with one node and one shard.
+{: .warning}
@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Personalize search ranking processor
+title: Personalize search ranking
 nav_order: 40
 has_children: false
 parent: Search processors

@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Rename field processor
+title: Rename field
 nav_order: 20
 has_children: false
 parent: Search processors

@@ -1,6 +1,6 @@
 ---
 layout: default
-title: Script processor
+title: Script
 nav_order: 30
 has_children: false
 parent: Search processors

@@ -13,9 +13,12 @@ Search processors can be of the following types:
 
 - [Search request processors](#search-request-processors)
 - [Search response processors](#search-response-processors)
+- [Search phase results processors](#search-phase-results-processors)
 
 ## Search request processors
 
+A search request processor takes a search request (the query and the metadata passed in the request) and performs an operation on the search request before submitting the search request to the index.
-A search request processor takes a search request (the query and the metadata passed in the request) and performs an operation on the search request before submitting the search request to the index.
+A search request processor performs an operation on a search request (the query and the metadata passed in the request) before submitting it to the index.
-A search request processor takes a search request (the query and the metadata passed in the request) and performs an operation on the search request before submitting the search request to the index.
+A search request processor performs an operation on a search request (the query and the metadata passed in the request) before submitting it to the index.
+
 The following table lists all supported search request processors.
 
 Processor | Description | Earliest available version
@@ -25,13 +28,25 @@ Processor | Description | Earliest available version
 
 ## Search response processors
 
+A search response processor performs an operation on the search response and returns a search response.
+
 The following table lists all supported search response processors.
 
 Processor | Description | Earliest available version
 :--- | :--- | :---
 [`rename_field`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rename-field-processor/)| Renames an existing field. | 2.8
 [`personalize_search_ranking`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/personalize-search-ranking/) | Uses [Amazon Personalize](https://aws.amazon.com/personalize/) to rerank search results (requires setting up the Amazon Personalize service). | 2.9
 
+## Search phase results processors
+
+A search phase results processor runs between search phases at the coordinating node level. It takes the results retrieved from one search phase and transforms them before passing them to the next search phase. 
-A search phase results processor runs between search phases at the coordinating node level. It takes the results retrieved from one search phase and transforms them before passing them to the next search phase. 
+A search phase results processor runs between search phases at the coordinating node level. It transforms the results retrieved from one search phase before passing them to the next search phase. 
-A search phase results processor runs between search phases at the coordinating node level. It takes the results retrieved from one search phase and transforms them before passing them to the next search phase. 
+A search phase results processor runs between search phases at the coordinating node level. It transforms the results retrieved from one search phase before passing them to the next search phase. 
+
+The following table lists all supported search request processors.
+
+Processor | Description | Earliest available version
+:--- | :--- | :---
+[`normalization_processor`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/script-processor/) | Intercepts the query phase results and normalizes and combines the document scores before passing the documents to the fetch phase. | 2.10
+
 ## Viewing available processor types
 
 You can use the Nodes Search Pipelines API to view the available processor types: