[BUG] [Segment Replication] Handle percolate queries

[dreamer-89]

Coming from discussion in opensearch-project/alerting#974 (comment), core needs to handle percolate queries for SEGMENT replication enabled indices.

CC @mch2 @anasalkouz

[anasalkouz]

@dreamer-89 , Not sure how percolate queries will be impacted by replication method. could you elaborate more?

[lezzago]

For percolate queries, we must initially index a document of the queries that we would want to execute on an index/indices. So the behavior here is we initially index a document of queries and then immediately run percolate query using the index storing the queries, which was just written to for adding the queries to run, against the data that we would want to query.

In the Alerting plugin this is impactful because when we run Document level monitors, we are updating an alerting index (we have a refresh immediate policy) that stores the queries we want to run and then immediately run the percolate query to query the data. This means we are immediately reading after writing for the alerting index. With the seg rep changes, the core needs to makes changes in the percolate query logic so it searches the primary shards of the index that stores the queries, so the most up to date queries are retrieved and the query would output the correct results.

[anasalkouz]

and what is the concern with the proposed solution by routing queries to the primary shard?

[lezzago]

Just want to call out for Segment Replication to be enabled by default, this issue would need to be fixed.

[anasalkouz]

For 2.10 SegRep is not enabled by default, and we will document this percolate query gap on SegRep documentation. We will track adding the ability to route percolate queries to primary separately and it should't be a blocker for 2.10.

[tlfeng]

The problem is caused by the current limitation which described in issue #8700
Currently the acceptable solution is the same as issue #8536 and #9563 .

Specifically, the solution here is to identify percolate query from the search request that comes into TransportSearchAction class and apply _primary to preference parameter, when segment replication is enabled and no other preference is assigned by the user.

OpenSearch/server/src/main/java/org/opensearch/action/search/TransportSearchAction.java

Line 464 in 1164221

searchRequest.preference(),

[anasalkouz]

I don't think we should route it to primary by default. We should keep this to user if they want to route to primary, we only need to give them the ability to choose per query.

[tlfeng]

There is a preference query parameter for Search request, see the docs.

When running the search request with preference=_primary parameter, it will perform the search only on primary shards. For example GET /my-index/_search?preference=_primary.
It can be applied to Percolate query, so I think it is a workaround to overcome the consistency limitation of segment replication. I need to double check whether it's effective on fetching the stored query.

To clarify the parameters, there are 2 preference parameter can be configured to Percolate query. One used to executing a get request to fetch the document to be percolated (docs) , and another one is the general parameter for all search requests. The preference to deal with this issue is the latter.

[mch2]

Proposing #10150 as an alternative here to updating the clients with prefer primary.

[mch2]

Alerting has updated to prefer primary opensearch-project/alerting#1205 - we don't have any special handling of percolate queries required.