Add '_name' field support to score functions and provide it back in explanation response

[reta]

Signed-off-by: Andriy Redko [email protected]

Description

The Opensearch supports _name property to tag for queries / filters [1] but those are never returned as part of Explanation description. The proposed change extends _name property support to scoring functions and embeds the _name into Explanation description if available.

• request / response with script function score

"functions": [
    {
      "script_score": {
        "script": {
          "lang": "painless",
          "source": "return doc['user_name'].contains('cash') ? 1 : 0;"            }
      },
      "weight": 65,
      // NEW: Function name 
      "_name": "func1"
    }, ...    
]

{
    "value": 0.0,
      // NEW: Function name is returned back
    "description": "script score function(_name: func1), computed with script:\"Script{type=inline, lang='painless', idOrCode='return doc['user_name'].contains('cash') ? 1 : 0;', options={}, params={}}\"",
    "details": [
        {
            "value": 1.0,
            "description": "_score: ",
            "details": [
                {
                    "value": 1.0,
                    "description": "*:*",
                    "details": []
                }
            ]
        }
    ]
}

• request / response with random function score

"functions": [
    {
        "random_score": {
           // NEW: Function name 
           "_name": "func2"
        }
    }, ...    
]

{
    "value": 0.38554674,
    // NEW: Function name is returned back
    "description": "random score function (seed: 738562412, field: null, _name: func2)",
    "details": []
} ...

• request / response with filter score function

"functions": [
    {
        "filter": {
            "terms": {
               // Query names are supported
                "_name": "terms_filter",
                "abc": [
                  "1"
                ]
              }
            },
        "weight": 35
    }, ...    
]

{
    "value": 1.0,
    // NEW: Query name is returned back
    "description": "match filter(_name: terms_filter): abc:{1}",
    "details": []
}

• request / response with function score query

 "function_score": {
      "query": {
        "match_all": {
            // Query names are supported
            "_name": "q1"
        }
      },
      ...
}

{
    "value": 1.0,
    "description": "_score: ",
    "details": [
        {
            "value": 1.0,
            // NEW: Query name is returned back
            "description": "*:* (_name: q1)",
            "details": []
        }
    ]
}

TODO:

• Cover all combination of queries / filters / functions
• Add more tests

[1] https://www.elastic.co/guide/en/elasticsearch/reference/7.10/query-dsl-bool-query.html#named-queries

Issues Resolved

Closes #1711

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-ci-bot]

Can one of the admins verify this patch?

[opensearch-ci-bot]

✅   Gradle Check success ee12649cee6467e46e7bc531e12c7751482f6077
Log 2786

Reports 2786

[opensearch-ci-bot]

✅   Gradle Check success 2ef9baed819221aa26af356c641412fc6515c11b
Log 2814

Reports 2814

[opensearch-ci-bot]

✅   Gradle Check success bc26b10086fe5cd288cb777663cf14c5be02370c
Log 2824

Reports 2824

[opensearch-ci-bot]

❌   Gradle Check failure 87a3632bc25be60125f752a2e4e8c968f8c935e2
Log 2886

Reports 2886

[opensearch-ci-bot]

❌   Gradle Check failure c9128647e1c6be0058b0975a833ae84f73f8cb28
Log 2887

Reports 2887

[reta]

x Gradle Check failure c912864 Log 2887

Reports 2887

@andrross CentOS again

| �[91mErrors during downloading metadata for repository 'appstream':
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml [Connection timed out after 30000 milliseconds]
|   - Curl error (56): Failure when receiving data from the peer for http://vault.centos.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml [Recv failure: Connection reset by peer]
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/AppStream/x86_64/os/repodata/repomd.xml [Connection timed out after 30001 milliseconds]

[opensearch-ci-bot]

✅   Gradle Check success 5458e1a22c01fa67293286ee7a4af302a77f0158
Log 2896

Reports 2896

[dblock]

Should we also be adding a _name field into _details? Or is that too function-specific?

{
    "value": 0.0,
      // NEW: Function name is returned back
    "description": "script score function(_name: func1), computed with script:\"Script{type=inline, lang='painless', idOrCode='return doc['user_name'].contains('cash') ? 1 : 0;', options={}, params={}}\"",
    "details": [
        {
            "value": 1.0,
            "description": "_score: ",
           // NEW:
            "_name": "func1",
            "details": [
                {
                    "value": 1.0,
                    "description": "*:*",
                    "details": []
                }
            ]
        }
    ]
}

{
    "value": 1.0,
    // NEW: Query name is returned back
    "description": "match filter(_name: terms_filter): abc:{1}",
    "details": {
           // NEW:
        "_name": "terms_filter"
    }
}

[reta]

Should we also be adding a _name field into _details? Or is that too function-specific?

{
    "value": 0.0,
      // NEW: Function name is returned back
    "description": "script score function(_name: func1), computed with script:\"Script{type=inline, lang='painless', idOrCode='return doc['user_name'].contains('cash') ? 1 : 0;', options={}, params={}}\"",
    "details": [
        {
            "value": 1.0,
            "description": "_score: ",
           // NEW:
            "_name": "func1",
            "details": [
                {
                    "value": 1.0,
                    "description": "*:*",
                    "details": []
                }
            ]
        }
    ]
}

{
    "value": 1.0,
    // NEW: Query name is returned back
    "description": "match filter(_name: terms_filter): abc:{1}",
    "details": {
           // NEW:
        "_name": "terms_filter"
    }
}

Thanks for looking @dblock, the fundamental problem right now is that we cannot extend the explanation easily [1], it is coming from Apache Lucene and is deeply integrated into its core (Weight, ...). The interim option is to include the _name into description whenever possible, but in some cases we cannot do that even that (fe BooleanWeight).

[1] https://issues.apache.org/jira/browse/LUCENE-10432

[reta]

Preserving for backward compatibility

[opensearch-ci-bot]

✅   Gradle Check success a12238c245037dad3fb073b233e130fdb791e3e1
Log 2938

Reports 2938

[opensearch-ci-bot]

✅   Gradle Check success 9a1ac65c74734ad58862516c40574203d1efe957
Log 2939

Reports 2939

[dblock]

@reta Thank you! Merging this feels like a no-regret move.
@lrynek please comment on this implementation?

[reta]

Thanks @dblock! It looks I covered all the cases (besides the core Lucene), @lrynek would it be helpful?

[lrynek]

@dblock @reta hi guys! 👋 Thank you for delivering it so fast! It surely helps and fits our needs at https:/DocPlanner ❤️ 💪 🎉

[reta]

@nknize @andrross could you please take a look guys, thank you!

[lrynek]

@reta does it mean that this PR will target next major (2.0.0) release of OpenSearch, right? 🤔

[reta]

@lrynek the way we structure the pull requests: main first, backport to 1.x after, if there are no major issues / objections, it could land in 1.3.0 (since this is not a breaking change).

[lrynek]

@reta thanks for explanation 👍 // nomen omen 😅

[opensearch-ci-bot]

❌   Gradle Check failure f136f54ff3e38bff1ffa93fd8230aaa092ff18d2
Log 2996

Reports 2996

[reta]

❌ Gradle Check failure f136f54 Log 2996

Reports 2996

* What went wrong:
Execution failed for task ':qa:translog-policy:v1.2.5#Step1OldClusterTest'.
> `cluster{:qa:translog-policy:v1.2.5}` failed to wait for cluster health yellow after 40 SECONDS
    IO error while waiting cluster
    503 Service Unavailable

[opensearch-ci-bot]

❌   Gradle Check failure 9a1ac65c74734ad58862516c40574203d1efe957
Log 2997

Reports 2997

[opensearch-ci-bot]

❌   Gradle Check failure f136f54ff3e38bff1ffa93fd8230aaa092ff18d2
Log 2998

Reports 2998

[dblock]

Something is going on here across multiple PRs.

REPRODUCE WITH: ./gradlew ':qa:full-cluster-restart:v1.3.0#oldClusterTest' --tests "org.opensearch.upgrades.FullClusterRestartIT.testSnapshotRestore" -Dtests.seed=1372DD30D319D963 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-TW -Dtests.timezone=Europe/Rome -Druntime.java=17

org.opensearch.upgrades.FullClusterRestartIT > testSnapshotRestore FAILED
    org.opensearch.client.WarningFailureException: method [PUT], host [http://127.0.0.1:37897], URI [/testsnapshotrestore], status line [HTTP/1.1 200 OK]
    {"acknowledged":true,"shards_acknowledged":true,"index":"testsnapshotrestore"}
        at __randomizedtesting.SeedInfo.seed([1372DD30D319D963:2C1737C919A4DE1C]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:976)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:956)
        at app//org.opensearch.test.rest.OpenSearchRestTestCase.createIndex(OpenSearchRestTestCase.java:952)
        at app//org.opensearch.upgrades.FullClusterRestartIT.testSnapshotRestore(FullClusterRestartIT.java:810)

Doesn't look like a fluke anymore.

[dblock]

start gradle check

[opensearch-ci-bot]

❌   Gradle Check failure d6ae4a0725960308df180d2307ffd5422c2a5563
Log 3001

Reports 3001

[opensearch-ci-bot]

❌   Gradle Check failure d6ae4a0725960308df180d2307ffd5422c2a5563
Log 3003

Reports 3003

[opensearch-ci-bot]

❌   Gradle Check failure cc3e322f835faa24361768d40de21b498d2b066e
Log 3023

Reports 3023

[opensearch-ci-bot]

❌   Gradle Check failure b0f14b6818db321348f52dc83d3736cf64cd1e33
Log 3025

Reports 3025

[dblock]

I merged #2339 that should fix this, rebase?

[opensearch-ci-bot]

✅   Gradle Check success 3eecdc4
Log 3031

Reports 3031

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-2244-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5f90227a05fec7fec03854867563652690e192e7
# Push it to GitHub
git push --set-upstream origin backport/backport-2244-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-2244-to-1.x.

[dblock]

@reta You'll need to backport this manually if you want it in 1.3.0.