[Segment Replication] Update flaky testOnNewCheckpointFromNewPrimaryCancelOngoingReplication unit test

[dreamer-89]

Manual backport of #4414 to 2.x

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2742/
• CommitID: 72a6b06

[dreamer-89]

Java heap space issue. Tracked in #3973

BUILD SUCCESSFUL in 39m 2s
2605 actionable tasks: 2591 executed, 2 from cache, 12 up-to-date

FAILURE: Build failed with an exception.

* What went wrong:
Java heap space

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2759/
• CommitID: 58a202e

[dreamer-89]

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark" -Dtests.seed=FDA33BAC00ED1B41 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=cs -Dtests.timezone=Asia/Krasnoyarsk -Druntime.java=17

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testRestoreSnapshotAllocationDoesNotExceedWatermark FAILED
    java.lang.AssertionError: 
    Expected: an empty collection
         but: <[[mdxatmiwym][0], node[9OQxqU_eTUaV--kxZPJt5g], [P], s[STARTED], a[id=3X3cxNoDTUuJb0-hskS8Tg]]>
        at __randomizedtesting.SeedInfo.seed([FDA33BAC00ED1B41:EF1B9DCD6B5404F9]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$testRestoreSnapshotAllocationDoesNotExceedWatermark$2(DiskThresholdDeciderIT.java:260)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1022)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark(DiskThresholdDeciderIT.java:260)
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.5.1-all/1ehga6e77gqps5uk2kc5kf1vc/gradle-7.5.1/lib/plugins/gradle-testing-****-7.5.1.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/2763/
• CommitID: 58a202e

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/2768/
• CommitID: d9860d5

[codecov-commenter]

Codecov Report

Merging #4425 (d9860d5) into 2.x (1f339e5) will decrease coverage by 0.06%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                2.x    #4425      +/-   ##
============================================
- Coverage     70.68%   70.62%   -0.07%     
+ Complexity    57195    57145      -50     
============================================
  Files          4585     4585              
  Lines        274494   274495       +1     
  Branches      40226    40226              
============================================
- Hits         194036   193858     -178     
- Misses        64278    64410     +132     
- Partials      16180    16227      +47     

Impacted Files	Coverage Δ
.../indices/replication/SegmentReplicationTarget.java	91.26% <100.00%> (ø)
...org/opensearch/index/shard/IndexShardTestCase.java	94.44% <100.00%> (+0.01%)	⬆️
...adonly/AddIndexBlockClusterStateUpdateRequest.java	0.00% <0.00%> (-75.00%)	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	9.75% <0.00%> (-58.54%)	⬇️
...ava/org/opensearch/action/NoSuchNodeException.java	0.00% <0.00%> (-50.00%)	⬇️
...ch/transport/ReceiveTimeoutTransportException.java	50.00% <0.00%> (-50.00%)	⬇️
.../opensearch/client/indices/CloseIndexResponse.java	42.50% <0.00%> (-48.75%)	⬇️
.../java/org/opensearch/search/dfs/AggregatedDfs.java	51.61% <0.00%> (-45.17%)	⬇️
...h/action/ingest/SimulateDocumentVerboseResult.java	60.71% <0.00%> (-39.29%)	⬇️
...indices/readonly/TransportAddIndexBlockAction.java	20.68% <0.00%> (-37.94%)	⬇️
... and 496 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.