[Remote Store] Add remote segment upload backpressure integ tests

[ashking94]

Description

This PR adds integration tests for Remote Segment Backpressure.

Related Issues

Resolves #7527

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ABORTED ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18108/
• CommitID: 80a47fb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Seeing flaky tests - #8059, #5957, #5031 & #8213

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/18149/
• CommitID: 824b23d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8197 (34ecc46) into main (0c7ba94) will decrease coverage by 0.18%.
The diff coverage is n/a.

❗ Current head 34ecc46 differs from pull request most recent head 6834632. Consider uploading reports for the commit 6834632 to get more accurate results

@@             Coverage Diff              @@
##               main    #8197      +/-   ##
============================================
- Coverage     70.96%   70.79%   -0.18%     
+ Complexity    56780    56545     -235     
============================================
  Files          4733     4719      -14     
  Lines        268140   267559     -581     
  Branches      39287    39206      -81     
============================================
- Hits         190285   189406     -879     
- Misses        61816    62192     +376     
+ Partials      16039    15961      -78     

see 498 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/18152/
• CommitID: e574297
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/18154/
• CommitID: 34ecc46

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18437/
• CommitID: 2fcc975
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18438/
• CommitID: ff24dd5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Build failed due to lucene version tests failing. Unrelated to this PR. Seeing similar failures in other PR builds as well - https://build.ci.opensearch.org/job/gradle-check/18436/

REPRODUCE WITH: ./gradlew ':qa:verify-version-constants:v2.9.0#integTest' --tests "org.opensearch.qa.verify_version_constants.VerifyVersionConstantsIT.testLuceneVersionConstant" -Dtests.seed=4F5BE3F1B1D72CEF -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=vi-VN -Dtests.timezone=Europe/Warsaw -Druntime.java=20

org.opensearch.qa.verify_version_constants.VerifyVersionConstantsIT > testLuceneVersionConstant FAILED
    java.lang.AssertionError: 
    Expected: <9.7.0>
         but: was <9.6.0>
        at __randomizedtesting.SeedInfo.seed([4F5BE3F1B1D72CEF:C23EDEAD96DFEFA6]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.qa.verify_version_constants.VerifyVersionConstantsIT.testLuceneVersionConstant(VerifyVersionConstantsIT.java:56)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18441/
• CommitID: 6834632
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Backport was merged.