Fix SegmentReplication flaky integ tests

[sachinpkale]

Description

• Posting the comment as it is: The getLatestSegmentInfosAndCheckpoint method is eventually invoked and failing with the call to getEngine().config().getCodec().getName() because the engine is already closed. The method already has logic to return empty if the shard is not open, but it is shut after this check. So we need to ensure the shard is not closed while this method is invoked or if we don't want to block shard close for this catch the error and gracefully return.
• Some of the tests in SegmentReplicationIT are flaky. This was observed while debugging flakiness in SegmentReplicationRemoteStoreIT which extends SegmentReplicationIT with remote store settings.
• Reproducing the flakiness is not straightforward and using the same seed will not guarantee of failure. The only way of reproducing the issue is to run the test multiple times.
• After inspecting the logs of the failure, we found that the two nodes in the cluster are acting as data as well as cluster manager nodes. As part of test flow, when we stop one of the nodes, it may result in quorum loss issue for the cluster given only 2 nodes.
• To avoid this issue, in this PR, we have explicitly added a cluster manager only node for each test. We have also made primary and replica as data only nodes to make the tests more deterministic.

Related Issues

• [BUG] org.opensearch.indices.replication.SegmentReplicationIT.testReplicationPostDeleteAndForceMerge is flaky #8100

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17872/
• CommitID: 8bc89f4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/17873/
• CommitID: 46ccc1c
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8134 (2417663) into main (90678c2) will increase coverage by 0.07%.
The diff coverage is 29.41%.

@@             Coverage Diff              @@
##               main    #8134      +/-   ##
============================================
+ Coverage     70.98%   71.06%   +0.07%     
- Complexity    56687    56737      +50     
============================================
  Files          4722     4722              
  Lines        267608   267619      +11     
  Branches      39216    39218       +2     
============================================
+ Hits         189973   190172     +199     
+ Misses        61582    61463     -119     
+ Partials      16053    15984      -69     

Impacted Files	Coverage Δ
...in/java/org/opensearch/index/shard/IndexShard.java	70.41% <29.41%> (-0.55%)	⬇️

... and 474 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17996/
• CommitID: af7644a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18030/
• CommitID: af7644a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

• URL: https://build.ci.opensearch.org/job/gradle-check/18079/
• CommitID: 2417663
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.