Add score normalization and combination documentation #4985

kolchfa-aws · 2023-09-07T18:23:52Z

Fixes #4351 #4526

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Fanit Kolchina <[email protected]>

navneet1v · 2023-09-07T20:07:58Z

_query-dsl/compound/hybrid.md

+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https:/opensearch-project/neural-search/issues/244). 
+{: .warning}


This is not an experimental feature now. We have fixed the flag.

Feature is enabled by default as Navneet mentioned, it's possible to disable it using a feature flag plugins.neural_search.hybrid_search_disabled (corresponding PR opensearch-project/neural-search#274)

navneet1v · 2023-09-07T20:09:05Z

_query-dsl/compound/hybrid.md

+
+Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.
+
+## Example


can we have a full example of how to do hybrid search here. Example including the creation of Search Pipeline and then the hybrid query

Added a link to the example in the normalization processor documentation so it can be maintained in one place.

navneet1v

Somewhere in the documentation we should mention about this blog: https://opensearch.org/blog/semantic-science-benchmarks/

to give science background about this query type.

navneet1v · 2023-09-07T20:12:32Z

_query-dsl/compound/hybrid.md

+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https:/opensearch-project/neural-search/issues/244). 
+{: .warning}
+
+Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.


Suggested change

Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.

Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries and calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.

The blog is mentioned in the normalization processor

Signed-off-by: Fanit Kolchina <[email protected]>

martin-gaievski · 2023-09-07T23:03:20Z

_search-plugins/search-pipelines/normalization-processor.md

+
+## Score normalization and combination
+
+Many applications require both keyword matching and semantic understanding. For example, BM25 accurately provides relevant search results for a query containing keywords, and neural networks perform well when a query requires natural language understanding. Thus, you might want to combine BM25 search results with the results of k-NN or neural search. However, BM25 and k-NN search use different scales to calculate relevance scores for the matching documents. Before combining the scores from multiple queries, it is necessary to normalize those scores so they are on the same scale. For further reading about score normalization and combination, including benchmarks and discussion of various techniques, see this [semantic search blog](https://opensearch.org/blog/semantic-science-benchmarks/).


it is necessary to normalize those scores

I'm not sure it's necessary, it's rather proved by experimental data that final results do have better metric on information retrieval. Not a strong opinion, but maybe worth checking if we can formulate this in more relaxed way.

martin-gaievski · 2023-09-07T23:05:00Z

_search-plugins/search-pipelines/normalization-processor.md

+
+Field | Data type | Description
+:--- | :--- | :---
+`normalization.technique` | String | The technique for normalizing scores. Valid values are `min_max`, `L2`. Optional. Default is `min_max`.


l2 is the valid option that is supported, all lowercase

martin-gaievski · 2023-09-07T23:12:07Z

_search-plugins/search-pipelines/normalization-processor.md

+`combination.parameters.weights` | Array of floating-point values | Specifies the weights to use for each query. Valid values are in the [0.0, 1.0] range and signify decimal percentages. The closer the weight is to 1.0, the more weight is given to a query. The number of values in the `weights` array must equal the number of queries. The sum of the values in the array must equal 1.0. Optional. If not provided, all queries are given equal weight.
+`tag` | String | The processor's identifier. Optional.
+`description` | String | A description of the processor. Optional.
+`ignore_failure` | Boolean | If `true`, OpenSearch [ignores a failure]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/index/#ignoring-processor-failures) of this processor and continues to run the remaining processors in the search pipeline. Optional. Default is `false`.


this setting is hardcoded to false, user-defined value is ignored

martin-gaievski · 2023-09-07T23:13:20Z

_search-plugins/search-pipelines/normalization-processor.md

+ "combination": { 
+ "technique" : "arithmetic_mean", 
+ "parameters" : { 
+ "weights" : [0.4, 0.7] 


this will fail as weight are not sum up to 1.0, values like [0.4, 0.6] will work

martin-gaievski · 2023-09-07T23:18:29Z

_query-dsl/compound/hybrid.md

+This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https:/opensearch-project/neural-search/issues/244). 
+{: .warning}


Feature is enabled by default as Navneet mentioned, it's possible to disable it using a feature flag plugins.neural_search.hybrid_search_disabled (corresponding PR opensearch-project/neural-search#274)

Signed-off-by: Fanit Kolchina <[email protected]>

martin-gaievski · 2023-09-11T16:29:43Z

_ml-commons-plugin/semantic-search.md

+ "only_run_on_ml_node": "false",
+ "model_access_control_enabled": "true",
+ "native_memory_threshold": "99",
+ "allow_registering_model_via_url": "true"


None of those settings are required for hybrid search to work, mainly they are for ml-commons to upload the model that we're using to create embeddings, in other words it's possible to make it run and not set any of that settings. Say if you're using one of pre-built models which makes sense for the simple example, then "allow_registering_model_via_url" is not needed at all. "native_memory_threshold" is needed to minimize chance of open circuit breaker when machine for node is not that powerful, "only_run_on_ml_node" is used if this is simple cluster without dedicated ml nodes, and nodes of other types can be used. "model_access_control_enabled" is also optional, I think if security plugin is enabled then it's not required. Maybe say that is one of simplest config that will work in local, and also put a reference to ml-common config docs

Thanks, reworded.

Signed-off-by: Fanit Kolchina <[email protected]>

natebower

LGTM!

Signed-off-by: Fanit Kolchina <[email protected]>

ylwu-amzn · 2023-09-13T16:51:20Z

_ml-commons-plugin/semantic-search.md

+You can choose to use another model:
+
+- One of the [pretrained language models provided by OpenSearch]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/).
+- Your own model. For instructions how to set up a custom model, see [Model serving framework]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).


Should we change Model serving framework to ML framework?

Suggested change

- Your own model. For instructions how to set up a custom model, see [Model serving framework]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).

- Your own model. For instructions how to set up a custom model, see [ML framework]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).

Already changed, thanks!

ylwu-amzn · 2023-09-13T16:55:34Z

_ml-commons-plugin/semantic-search.md

+ </summary>
+ {: .text-delta}
+
+Search for the newly created model by providing its ID in the request:


I think we can just use get model API to get the newly created model by model id

Get the newly created model by providing its ID in the request:

ylwu-amzn · 2023-09-13T16:56:08Z

_ml-commons-plugin/semantic-search.md

+Search for the newly created model by providing its ID in the request:
+
+```json
+POST /_plugins/_ml/models/_search/aVeif4oB5Vm0Tdw8zYO2


Suggested change

POST /_plugins/_ml/models/_search/aVeif4oB5Vm0Tdw8zYO2

POST /_plugins/_ml/models/aVeif4oB5Vm0Tdw8zYO2

ylwu-amzn · 2023-09-13T16:56:48Z

_ml-commons-plugin/semantic-search.md

+The response contains the model:
+
+```json
+{


This response is search response, suggest change to get model response to match

ylwu-amzn · 2023-09-13T16:57:51Z

_ml-commons-plugin/semantic-search.md

+}
+```
+
+For more information, see [Model serving framework]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).


We renamed model serving framework as ML framework. How about change to ML framework?

ylwu-amzn · 2023-09-13T16:58:33Z

_ml-commons-plugin/semantic-search.md

+Search for the deployed model by providing its ID in the request:
+
+```json
+POST /_plugins/_ml/models/_search


How about just use get model API, much simpler

You should also change the response JSON

Signed-off-by: Fanit Kolchina <[email protected]>

dtaivpp

Overall really excited about this tutorial!

dtaivpp · 2023-09-13T18:51:39Z

_ml-commons-plugin/semantic-search.md

+It's helpful to understand the following terms before starting this tutorial:
+
+- _Semantic search_: Employs neural search in order to determine the intention of the user's query in the search context and improve search relevance. 
+- _Neural search_: When indexing documents containing text, neural search uses language models to generate vector embeddings from that text. When you then use a _neural query_, the query text is passed through a language model, and the resulting vector embeddings are compared with the document text vector embeddings to find the most relevant results.


Could we maybe use a visualization here to show how the workflow has changed? I think this may be hard for some people to understand even after reading.

@dtaivpp I'm not sure I understand what you mean by "workflow has changed." Do you mean provide a diagram of what happens when you generate embeddings and use them for search? I can do that.

dtaivpp · 2023-09-13T18:57:18Z

_ml-commons-plugin/semantic-search.md

+For a more advanced setup, note the following requirements:
+
+- To register a custom model, you need to specify an additional `"allow_registering_model_via_url": "true"` cluster setting. 
+- On clusters with dedicated ML nodes, you may want to specify `"only_run_on_ml_node": "true"` for improved performance. 


I think we should call out that it's best practice to separate the workloads by having dedicated ML nodes. Otherwise they could end up really hurting the performance of non-vector queries.

dtaivpp · 2023-09-13T19:02:20Z

_ml-commons-plugin/semantic-search.md

+
+### Step 1(a): Choose a language model
+
+For this tutorial, you'll use the [DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert) model from Hugging Face. It is one of the pretrained sentence transformer models available in OpenSearch. You'll need the name, version, and dimension of the model to register it. You can find this information in the [pretrained model table]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sentence-transformers) by selecting the `config_url` link corresponding to the model's TorchScript artifact:


If we have any reason this model was picked we should call that out here. That way others can find a model that suites them.

@martin-gaievski Do we have any reason this particular model was chosen?

For purpose of testing Hybrid search I was following previous blog from science team on building semantic search, they have mentioned TAS-B as a showing one of the best results among other models. Plus it's supported as one of pre-trained models by our ml-commons https://opensearch.org/docs/latest/ml-commons-plugin/pretrained-models/#sentence-transformers

dtaivpp · 2023-09-13T19:10:51Z

_ml-commons-plugin/semantic-search.md

+You can also receive statistics for all deployed models in your cluster by sending a Models Profile API request:
+
+```json
+GET /_plugins/_ml/profile/models


Is this a new API for 2.10?

No, it's not new

Ah totally had missed that in the docs. Cool!

I'll add a link so people can find the docs :)

We have one setting to control how many request the profile API will monitor. By default it only monitor last 100 requests.
Set the value as 0 will clean up all monitoring requests.

You can update the setting like this

PUT _cluster/settings { "persistent" : { "plugins.ml_commons.monitoring_request_count" : 1000000 } }

dtaivpp · 2023-09-13T19:14:38Z

_ml-commons-plugin/semantic-search.md

+
+## Step 2: Ingest data with neural search
+
+Neural search uses a language model to transform text into vector embeddings. During ingestion, neural search creates vector embeddings for the text fields in the request. During search, you can generate vector embeddings for the query text by applying the same model, allowing you to perform vector similarity search on the documents.


For this part: allowing you to perform vector similarity search on the documents should we call out that this is essentially what semantic search is is vector similarity using a sentence transformer that produces semantic embeddings.

Well.... you kinda have to provide the model in your query so you can do a similarity search yourself, right? So I think it's fine as is because this is more low-level so to speak and just lists the process. I also wanted to specifically call out neural search here and not semantic because this is the feature that we'll be using here. And semantic search is more like a conceptual term.

dtaivpp · 2023-09-13T19:17:37Z

_ml-commons-plugin/semantic-search.md

+
+### Step 2(a): Create an ingest pipeline for neural search
+
+The first step in setting up [neural search]({{site.url}}{{site.baseurl}}/search-plugins/neural-search/) is to create an [ingest pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/index/). The ingest pipeline will contain one processor: a task that transforms document fields. For neural search, you'll need to set up a `text_embedding` processor that takes in text and creates vector embeddings from that text. You'll need the `model_id` of the model you set up in the previous section and a `field_map`, which specifies the name of the field from which to take the text (`text`) and the name of the field in which to record embeddings (`passage_embedding`):


Maybe we should rephrase this? I feel like this isn't the first step as deploying a model was our first few steps. Maybe we can say, "Now that we have a model we can use that to configure neural search."

dtaivpp · 2023-09-13T19:35:44Z

_ml-commons-plugin/semantic-search.md

+
+You'll use the [`hybrid` query]({{site.url}}{{site.baseurl}}/query-dsl/compound/hybrid/) to combine the `match` and `neural` query clauses. Make sure to apply the previously created `nlp-search-pipeline` to the request in the query parameter:
+
+```json


Hey should we also pitch search templates here so that we dont have to worry about having the model id checked into code?

Sure, let me add it as an "advanced" note.

Signed-off-by: Fanit Kolchina <[email protected]>

natebower · 2023-09-15T11:58:39Z

_ml-commons-plugin/semantic-search.md

@@ -943,6 +974,10 @@ PUT /my-nlp-index/_settings

 You can now experiment with different weights, normalization techniques, and combination techniques. For more information, see the [`normalization_processor`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/normalization-processor/) and [`hybrid` query]({{site.url}}{{site.baseurl}}/query-dsl/compound/hybrid/) documentation.

+#### Advanced
+
+You can parametrize the search by using search templates, hiding implementation details and reducing the number of nested levels and thus the query complexity. For more information, see [search templates]({{site.url}}{{site.baseurl}}/search-plugins/search-template/).


"parameterize". "...by using search templates, hiding implementation details, or reducing the number of nested levels, thus reducing the query complexity"?

natebower · 2023-09-15T12:01:59Z

_ml-commons-plugin/semantic-search.md

@@ -97,15 +103,15 @@ Neural search requires a language model in order to generate vector embeddings f

 ### Step 1(a): Choose a language model

-For this tutorial, you'll use the [DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert) model from Hugging Face. It is one of the pretrained sentence transformer models available in OpenSearch. You'll need the name, version, and dimension of the model to register it. You can find this information in the [pretrained model table]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sentence-transformers) by selecting the `config_url` link corresponding to the model's TorchScript artifact:
+For this tutorial, you'll use the [DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert) model from Hugging Face. It is one of the pretrained sentence transformer models available in OpenSearch that has shown one of the best results in benchmarking tests (for details, see [this blog](https://opensearch.org/blog/semantic-science-benchmarks/)). You'll need the name, version, and dimension of the model to register it. You can find this information in the [pretrained model table]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sentence-transformers) by selecting the `config_url` link corresponding to the model's TorchScript artifact:


"some" instead of "one" of the best results. see this blog "post".

Signed-off-by: Fanit Kolchina <[email protected]>

ylwu-amzn · 2023-09-15T19:12:56Z

_ml-commons-plugin/semantic-search.md

+```json
+POST /_plugins/_ml/models/_register
+{
+ "name": "huggingface/sentence-transformers/msmarco-distilbert-base-tas-b",


I don't see url parameter in this request body, can you verify if this can work?

This is one of the pretrained models. It worked when I ran the request.

martin-gaievski · 2023-09-15T23:33:27Z

_search-plugins/search-pipelines/normalization-processor.md

+```
+{% include copy-curl.html %}
+
+For more information, see [Hybrid query]({{site.url}}{{site.baseurl}}/query-dsl/compound/hybrid/).


@kolchfa-aws Can we please add section (or include to one of existing sections) below information:

Search tuning

We have identified some recommendation on tuning search relevancy

Increase number of samples

If you're not seeing some results that are expected from hybrid query, that can be due to smallest size for each of the sub-queries. Only results returned by each individual sub-query are passed to the normalization processor, it does not perform additional sampling.
From our experiments we have found that size in range of [100,200] works best for datasets for up to 10M documents if we're using nDCG@10 to measure quality of information retrieval. Higher values of size do not give gain in search results quality. At the same time such higher values increase search latency so we do not recommend to go beyond recommended values.

Signed-off-by: Fanit Kolchina <[email protected]>

natebower

@kolchfa-aws Two last comments. Thanks!

natebower · 2023-09-18T19:49:23Z

_ml-commons-plugin/api.md

@@ -518,8 +518,24 @@ The API returns the following:

 ## Profile

-The profile operation returns runtime information on ML tasks and models. The profile operation can help debug issues with models at runtime.
+The profile operation returns runtime information on ML tasks and models. The profile operation can help debug issues with models at runtime. 


runtime information "about" instead of "on"? "The profile operation can help debug model issues at runtime."

natebower · 2023-09-18T19:52:05Z

_search-plugins/search-pipelines/normalization-processor.md

+
+To improve search relevance, we recommend increasing the sample size.
+
+If you don't see some results you expect the hybrid query to return, it can be because the subqueries return too few documents. The `normalization_processor` only transforms the results returned by each subquery; it does not perform any additional sampling. During our experiments, we used [nDCG@10](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) to measure quality of information retrieval depending on the number of documents returned (the size). We have found that size in the [100, 200] range works best for datasets of up to 10M documents. We do not recommend increasing the size beyond the recommended values because higher values of size do not improve search relevance but increase search latency.


"If the hybrid query does not return some expected results, it may be because..." Penultimate sentence: "a" before "size". Last sentence: "higher size values" instead of "higher values of size".

Signed-off-by: Fanit Kolchina <[email protected]>

…ject#4985) * Add search phase results processor Signed-off-by: Fanit Kolchina <[email protected]> * Add hybrid query Signed-off-by: Fanit Kolchina <[email protected]> * Normalization processor additions Signed-off-by: Fanit Kolchina <[email protected]> * Add more details Signed-off-by: Fanit Kolchina <[email protected]> * Continue writing Signed-off-by: Fanit Kolchina <[email protected]> * Add more query then fetch details and diagram Signed-off-by: Fanit Kolchina <[email protected]> * Small rewording Signed-off-by: Fanit Kolchina <[email protected]> * Leaner left nav headers Signed-off-by: Fanit Kolchina <[email protected]> * Tech review feedback Signed-off-by: Fanit Kolchina <[email protected]> * Add semantic search tutorial Signed-off-by: Fanit Kolchina <[email protected]> * Reworded prerequisites Signed-off-by: Fanit Kolchina <[email protected]> * Removed comma Signed-off-by: Fanit Kolchina <[email protected]> * Rewording advanced prerequisites Signed-off-by: Fanit Kolchina <[email protected]> * Changed searching for ML model to shorter request Signed-off-by: Fanit Kolchina <[email protected]> * Update task type in register model response Signed-off-by: Fanit Kolchina <[email protected]> * Changing example Signed-off-by: Fanit Kolchina <[email protected]> * Added huggingface prefix to model names Signed-off-by: Fanit Kolchina <[email protected]> * Change example responses Signed-off-by: Fanit Kolchina <[email protected]> * Added note about huggingface prefix Signed-off-by: Fanit Kolchina <[email protected]> * Update _ml-commons-plugin/semantic-search.md Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Implemented doc review comments Signed-off-by: Fanit Kolchina <[email protected]> * List weights under parameters Signed-off-by: Fanit Kolchina <[email protected]> * Remove one-shard warning for normalization processor Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Implemented editorial comments Signed-off-by: Fanit Kolchina <[email protected]> * Change links Signed-off-by: Fanit Kolchina <[email protected]> * More editorial feedback Signed-off-by: Fanit Kolchina <[email protected]> * Change model-serving framework to ML framework Signed-off-by: Fanit Kolchina <[email protected]> * Use get model API to check model status Signed-off-by: Fanit Kolchina <[email protected]> * Implemented tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Added neural search description and diagram Signed-off-by: Fanit Kolchina <[email protected]> * More editorial comments Signed-off-by: Fanit Kolchina <[email protected]> * Add link to profile API Signed-off-by: Fanit Kolchina <[email protected]> * Addressed more tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Implemented editorial comments on changes Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

* Add search phase results processor Signed-off-by: Fanit Kolchina <[email protected]> * Add hybrid query Signed-off-by: Fanit Kolchina <[email protected]> * Normalization processor additions Signed-off-by: Fanit Kolchina <[email protected]> * Add more details Signed-off-by: Fanit Kolchina <[email protected]> * Continue writing Signed-off-by: Fanit Kolchina <[email protected]> * Add more query then fetch details and diagram Signed-off-by: Fanit Kolchina <[email protected]> * Small rewording Signed-off-by: Fanit Kolchina <[email protected]> * Leaner left nav headers Signed-off-by: Fanit Kolchina <[email protected]> * Tech review feedback Signed-off-by: Fanit Kolchina <[email protected]> * Add semantic search tutorial Signed-off-by: Fanit Kolchina <[email protected]> * Reworded prerequisites Signed-off-by: Fanit Kolchina <[email protected]> * Removed comma Signed-off-by: Fanit Kolchina <[email protected]> * Rewording advanced prerequisites Signed-off-by: Fanit Kolchina <[email protected]> * Changed searching for ML model to shorter request Signed-off-by: Fanit Kolchina <[email protected]> * Update task type in register model response Signed-off-by: Fanit Kolchina <[email protected]> * Changing example Signed-off-by: Fanit Kolchina <[email protected]> * Added huggingface prefix to model names Signed-off-by: Fanit Kolchina <[email protected]> * Change example responses Signed-off-by: Fanit Kolchina <[email protected]> * Added note about huggingface prefix Signed-off-by: Fanit Kolchina <[email protected]> * Update _ml-commons-plugin/semantic-search.md Co-authored-by: Naarcha-AWS <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Implemented doc review comments Signed-off-by: Fanit Kolchina <[email protected]> * List weights under parameters Signed-off-by: Fanit Kolchina <[email protected]> * Remove one-shard warning for normalization processor Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> * Implemented editorial comments Signed-off-by: Fanit Kolchina <[email protected]> * Change links Signed-off-by: Fanit Kolchina <[email protected]> * More editorial feedback Signed-off-by: Fanit Kolchina <[email protected]> * Change model-serving framework to ML framework Signed-off-by: Fanit Kolchina <[email protected]> * Use get model API to check model status Signed-off-by: Fanit Kolchina <[email protected]> * Implemented tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Added neural search description and diagram Signed-off-by: Fanit Kolchina <[email protected]> * More editorial comments Signed-off-by: Fanit Kolchina <[email protected]> * Add link to profile API Signed-off-by: Fanit Kolchina <[email protected]> * Addressed more tech review comments Signed-off-by: Fanit Kolchina <[email protected]> * Implemented editorial comments on changes Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

kolchfa-aws added 6 commits August 28, 2023 13:58

Add search phase results processor

6ab7d57

Signed-off-by: Fanit Kolchina <[email protected]>

Add hybrid query

79e9597

Signed-off-by: Fanit Kolchina <[email protected]>

Normalization processor additions

69d4274

Signed-off-by: Fanit Kolchina <[email protected]>

Add more details

06dcb26

Signed-off-by: Fanit Kolchina <[email protected]>

Continue writing

f0d1667

Signed-off-by: Fanit Kolchina <[email protected]>

Add more query then fetch details and diagram

0ff381f

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws requested review from cwillum, hdhalter, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners September 7, 2023 18:23

kolchfa-aws added 2 commits September 7, 2023 14:26

Small rewording

32f7a6e

Signed-off-by: Fanit Kolchina <[email protected]>

Leaner left nav headers

8b0bb3d

Signed-off-by: Fanit Kolchina <[email protected]>

navneet1v reviewed Sep 7, 2023

View reviewed changes

Tech review feedback

76e5164

Signed-off-by: Fanit Kolchina <[email protected]>

martin-gaievski reviewed Sep 7, 2023

View reviewed changes

kolchfa-aws self-assigned this Sep 8, 2023

kolchfa-aws added v2.10.0 release-notes PR: Include this PR in the automated release notes labels Sep 8, 2023

Add semantic search tutorial

2fe3464

Signed-off-by: Fanit Kolchina <[email protected]>

hdhalter added the 3 - Tech review PR: Tech review in progress label Sep 11, 2023

martin-gaievski reviewed Sep 11, 2023

View reviewed changes

kolchfa-aws added 2 commits September 11, 2023 13:20

Reworded prerequisites

c353572

Signed-off-by: Fanit Kolchina <[email protected]>

Removed comma

9cff096

Signed-off-by: Fanit Kolchina <[email protected]>

natebower approved these changes Sep 13, 2023

View reviewed changes

More editorial feedback

0c7b587

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws added 6 - Done but waiting to merge PR: The work is done and ready to merge and removed 3 - Tech review PR: Tech review in progress labels Sep 13, 2023

ylwu-amzn reviewed Sep 13, 2023

View reviewed changes

kolchfa-aws added 2 commits September 13, 2023 13:03

Change model-serving framework to ML framework

6d48caf

Signed-off-by: Fanit Kolchina <[email protected]>

Use get model API to check model status

838b42f

Signed-off-by: Fanit Kolchina <[email protected]>

dtaivpp reviewed Sep 13, 2023

View reviewed changes

kolchfa-aws added 2 commits September 13, 2023 17:59

Implemented tech review comments

9ead908

Signed-off-by: Fanit Kolchina <[email protected]>

Added neural search description and diagram

8f292f1

Signed-off-by: Fanit Kolchina <[email protected]>

natebower reviewed Sep 15, 2023

View reviewed changes

kolchfa-aws added 2 commits September 15, 2023 09:59

More editorial comments

6fd7468

Signed-off-by: Fanit Kolchina <[email protected]>

Add link to profile API

20cb3df

Signed-off-by: Fanit Kolchina <[email protected]>

ylwu-amzn reviewed Sep 15, 2023

View reviewed changes

martin-gaievski reviewed Sep 15, 2023

View reviewed changes

Addressed more tech review comments

0c3f589

Signed-off-by: Fanit Kolchina <[email protected]>

martin-gaievski approved these changes Sep 18, 2023

View reviewed changes

natebower approved these changes Sep 18, 2023

View reviewed changes

Implemented editorial comments on changes

76036c4

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws merged commit f5b8ff7 into main Sep 22, 2023
4 checks passed

Naarcha-AWS deleted the score-combination branch March 28, 2024 23:23

		This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https:/opensearch-project/neural-search/issues/244).
		{: .warning}


		Use a hybrid query to combine relevance scores from multiple queries into one score for a given document. A hybrid query contains a list of one or more queries, which are run in parallel at the data node level. It calculates document scores at the shard level independently for each subquery. The subquery rewriting is done at the coordinating node level to avoid duplicate computations.

		## Example


		## Score normalization and combination

		Many applications require both keyword matching and semantic understanding. For example, BM25 accurately provides relevant search results for a query containing keywords, and neural networks perform well when a query requires natural language understanding. Thus, you might want to combine BM25 search results with the results of k-NN or neural search. However, BM25 and k-NN search use different scales to calculate relevance scores for the matching documents. Before combining the scores from multiple queries, it is necessary to normalize those scores so they are on the same scale. For further reading about score normalization and combination, including benchmarks and discussion of various techniques, see this [semantic search blog](https://opensearch.org/blog/semantic-science-benchmarks/).

	- Your own model. For instructions how to set up a custom model, see [Model serving framework]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).
	- Your own model. For instructions how to set up a custom model, see [ML framework]({{site.url}}{{site.baseurl}}/ml-commons-plugin/ml-framework/).

	POST /_plugins/_ml/models/_search/aVeif4oB5Vm0Tdw8zYO2
	POST /_plugins/_ml/models/aVeif4oB5Vm0Tdw8zYO2


		### Step 1(a): Choose a language model

		For this tutorial, you'll use the [DistilBERT](https://huggingface.co/docs/transformers/model_doc/distilbert) model from Hugging Face. It is one of the pretrained sentence transformer models available in OpenSearch. You'll need the name, version, and dimension of the model to register it. You can find this information in the [pretrained model table]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sentence-transformers) by selecting the `config_url` link corresponding to the model's TorchScript artifact:


		## Step 2: Ingest data with neural search

		Neural search uses a language model to transform text into vector embeddings. During ingestion, neural search creates vector embeddings for the text fields in the request. During search, you can generate vector embeddings for the query text by applying the same model, allowing you to perform vector similarity search on the documents.


		### Step 2(a): Create an ingest pipeline for neural search

		The first step in setting up [neural search]({{site.url}}{{site.baseurl}}/search-plugins/neural-search/) is to create an [ingest pipeline]({{site.url}}{{site.baseurl}}/api-reference/ingest-apis/index/). The ingest pipeline will contain one processor: a task that transforms document fields. For neural search, you'll need to set up a `text_embedding` processor that takes in text and creates vector embeddings from that text. You'll need the `model_id` of the model you set up in the previous section and a `field_map`, which specifies the name of the field from which to take the text (`text`) and the name of the field in which to record embeddings (`passage_embedding`):


		You'll use the [`hybrid` query]({{site.url}}{{site.baseurl}}/query-dsl/compound/hybrid/) to combine the `match` and `neural` query clauses. Make sure to apply the previously created `nlp-search-pipeline` to the request in the query parameter:

		```json


		To improve search relevance, we recommend increasing the sample size.

		If you don't see some results you expect the hybrid query to return, it can be because the subqueries return too few documents. The `normalization_processor` only transforms the results returned by each subquery; it does not perform any additional sampling. During our experiments, we used [nDCG@10](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) to measure quality of information retrieval depending on the number of documents returned (the size). We have found that size in the [100, 200] range works best for datasets of up to 10M documents. We do not recommend increasing the size beyond the recommended values because higher values of size do not improve search relevance but increase search latency.

Add score normalization and combination documentation #4985

Add score normalization and combination documentation #4985

Conversation

kolchfa-aws commented Sep 7, 2023

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

navneet1v left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

dtaivpp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn Sep 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Search tuning

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn Sep 13, 2023 •

edited

Loading

ylwu-amzn Sep 13, 2023 •

edited

Loading

ylwu-amzn Sep 15, 2023 •

edited

Loading