Fix Flaky Tests testReplicaThreadedUpdateToShardLimitsAndRejections and testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections #5439

Rishikesh1159 · 2022-12-02T23:57:53Z

Signed-off-by: Rishikesh1159 [email protected]

Description

This PR fixes flaky tests #4212 and #5157 by adding conditional/optional null check instead of compulsory null check. More details can be found in this comment

Issues Resolved

Resolves #4212 and #5157

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2022-12-03T00:15:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7557/
CommitID: 5812e6c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2022-12-03T00:25:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/7558/
CommitID: 46dd09a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

andrross · 2022-12-03T00:30:29Z

server/src/test/java/org/opensearch/index/ShardIndexingPressureConcurrentExecutionTests.java

@@ -269,7 +269,11 @@ public void testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections() th

 nodeStats = shardIndexingPressure.stats();
 IndexingPressurePerShardStats shardStoreStats = shardIndexingPressure.shardStats().getIndexingPressureShardStats(shardId1);
- assertNull(shardStoreStats);
+ if (rejectionCount.get() == NUM_THREADS) {


Should add a comment here explaining why there is a conditional check. Something like

// If rejection count equals NUM_THREADS that means rejections happened // until the last request, then we'll get shardStoreStats which was updated on // the last request. In other cases, the shardStoreStats simply moves to the // cold store and null is returned.

Sure, l will add it.

github-actions · 2022-12-03T00:51:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/7559/
CommitID: ae55b27

codecov-commenter · 2022-12-03T00:53:13Z

Codecov Report

Merging #5439 (ae55b27) into main (31bcda6) will decrease coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #5439      +/-   ##
============================================
- Coverage     71.01%   70.89%   -0.12%     
+ Complexity    58149    58083      -66     
============================================
  Files          4711     4711              
  Lines        277573   277573              
  Branches      40180    40180              
============================================
- Hits         197122   196789     -333     
- Misses        64293    64651     +358     
+ Partials      16158    16133      -25

Impacted Files	Coverage Δ
...adonly/AddIndexBlockClusterStateUpdateRequest.java	`0.00% <0.00%> (-75.00%)`	⬇️
...pensearch/client/cluster/RemoteConnectionInfo.java	`0.00% <0.00%> (-73.18%)`	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	`9.75% <0.00%> (-58.54%)`	⬇️
...a/org/opensearch/client/cluster/ProxyModeInfo.java	`0.00% <0.00%> (-55.00%)`	⬇️
...n/admin/indices/readonly/AddIndexBlockRequest.java	`17.85% <0.00%> (-53.58%)`	⬇️
...a/org/opensearch/client/cluster/SniffModeInfo.java	`0.00% <0.00%> (-52.95%)`	⬇️
.../java/org/opensearch/node/NodeClosedException.java	`50.00% <0.00%> (-50.00%)`	⬇️
...regations/metrics/AbstractHyperLogLogPlusPlus.java	`51.72% <0.00%> (-44.83%)`	⬇️
...pensearch/indices/breaker/CircuitBreakerStats.java	`27.77% <0.00%> (-41.67%)`	⬇️
...indices/readonly/TransportAddIndexBlockAction.java	`20.68% <0.00%> (-41.38%)`	⬇️
... and 472 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Rishikesh1159 <[email protected]>

github-actions · 2022-12-03T01:30:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/7563/
CommitID: bdcefdb
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <[email protected]> (cherry picked from commit 0bd3141) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <[email protected]> (cherry picked from commit 0bd3141) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <[email protected]>

…ature/identity (#5581) * Fix flaky ShardIndexingPressureConcurrentExecutionTests (#5439) Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <[email protected]> * Fix bwc for cluster manager throttling settings (#5305) Signed-off-by: Dhwanil Patel <[email protected]> * Update ingest-attachment plugin dependencies: Apache Tika 3.6.0, Apache Mime4j 0.8.8, Apache Poi 5.2.3, Apache PdfBox 2.0.27 (#5448) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> * Enhance CheckpointState to support no-op replication (#5282) * CheckpointState enhanced to support no-op replication Signed-off-by: Ashish Singh <[email protected]> Co-authored-by: Bukhtawar Khan<[email protected]> * [BUG] org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT/test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} flaky: randomizing basePath (#5482) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> * [Bug] fix case sensitivity for wildcard queries (#5462) Fixes the wildcard query to not normalize the pattern when case_insensitive is set by the user. This is achieved by creating a new normalizedWildcardQuery method so that query_string queries (which do not support case sensitivity) can still normalize the pattern when the default analyzer is used; maintaining existing behavior. Signed-off-by: Nicholas Walter Knize <[email protected]> * Support OpenSSL Provider with default Netty allocator (#5460) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> * Revert "build no-jdk distributions as part of release build (#4902)" (#5465) This reverts commit 8c9ca4e. It seems that this wasn't entirely the correct way and is currently blocking us from removing the `build.sh` from the `opensearch-build` repository (i.e. this `build.sh` here is not yet being used). See the discussion in opensearch-project/opensearch-build#2835 for further details. Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]> * Add max_shard_size parameter for Shrink API (fix supported version after backport) (#5503) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> * Sync CODEOWNERS with MAINTAINERS. (#5501) Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> * Added jackson dependency to server (#5366) * Added jackson dependency to server Signed-off-by: Ryan Bogan <[email protected]> * Updated CHANGELOG Signed-off-by: Ryan Bogan <[email protected]> * Update build.gradle files Signed-off-by: Ryan Bogan <[email protected]> * Add RuntimePermission to fix errors Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> * Fix flaky test BulkIntegrationIT.testDeleteIndexWhileIndexing (#5491) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Poojita Raj <[email protected]> * Add release notes for 2.4.1 (#5488) Signed-off-by: Xue Zhou <[email protected]> Signed-off-by: Xue Zhou <[email protected]> * Properly skip OnDemandBlockSnapshotIndexInputTests.testVariousBlockSize on Windows. (#5511) PR #5397 skipped this test in @before block but still frequently throws a TestCouldNotBeSkippedException. This is caused by the after block still executing and throwing an exception while cleaning the directory created at the path in @before. Moving the assumption to the individual test prevents this exception by ensuring the path exists. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> * Merge first batch of feature/extensions into main (#5347) * Merge first batch of feature/extensions into main Signed-off-by: Ryan Bogan <[email protected]> * Fixed CHANGELOG Signed-off-by: Ryan Bogan <[email protected]> * Fixed newline errors Signed-off-by: Ryan Bogan <[email protected]> * Renaming and CHANGELOG fixes Signed-off-by: Ryan Bogan <[email protected]> * Refactor extension loading into private method Signed-off-by: Ryan Bogan <[email protected]> * Removed skipValidation and added connectToExtensionNode method Signed-off-by: Ryan Bogan <[email protected]> * Remove unnecessary feature flag calls Signed-off-by: Ryan Bogan <[email protected]> * Renaming and exception handling Signed-off-by: Ryan Bogan <[email protected]> * Change latches to CompletableFuture Signed-off-by: Ryan Bogan <[email protected]> * Removed unnecessary validateSettingKey call Signed-off-by: Ryan Bogan <[email protected]> * Fix azure-core dependency Signed-off-by: Ryan Bogan <[email protected]> * Update SHAs Signed-off-by: Ryan Bogan <[email protected]> * Remove unintended dependency changes Signed-off-by: Ryan Bogan <[email protected]> * Removed dynamic settings regitration, removed info() method, and added NoopExtensionsManager Signed-off-by: Ryan Bogan <[email protected]> * Add javadoc Signed-off-by: Ryan Bogan <[email protected]> * Fixed spotless failure Signed-off-by: Ryan Bogan <[email protected]> * Removed NoopExtensionsManager Signed-off-by: Ryan Bogan <[email protected]> * Added functioning NoopExtensionsManager Signed-off-by: Ryan Bogan <[email protected]> * Added missing javadoc Signed-off-by: Ryan Bogan <[email protected]> * Remove forbiddenAPI Signed-off-by: Ryan Bogan <[email protected]> * Fix spotless Signed-off-by: Ryan Bogan <[email protected]> * Change logger.info to logger.error in handleException Signed-off-by: Ryan Bogan <[email protected]> * Fix ExtensionsManagerTests Signed-off-by: Ryan Bogan <[email protected]> * Removing unrelated change Signed-off-by: Ryan Bogan <[email protected]> * Update SHAs Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> * Bump commons-compress from 1.21 to 1.22 (#5520) Bumps commons-compress from 1.21 to 1.22. --- updated-dependencies: - dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [Segment Replication] Trigger a round of replication for replica shards during peer recovery when segment replication is enabled (#5332) * Fix new added replica shards falling behind primary. Signed-off-by: Rishikesh1159 <[email protected]> * Trigger a round of replication during peer recovery when segment replication is enabled. Signed-off-by: Rishikesh1159 <[email protected]> * Remove unnecessary start replication overloaded method. Signed-off-by: Rishikesh1159 <[email protected]> * Add test for failure case and refactor some code. Signed-off-by: Rishikesh1159 <[email protected]> * Apply spotless check. Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments on the PR. Signed-off-by: Rishikesh1159 <[email protected]> * Remove unnecessary condition check. Signed-off-by: Rishikesh1159 <[email protected]> * Apply spotless check. Signed-off-by: Rishikesh1159 <[email protected]> * Add step listeners to resolve forcing round of segment replication. Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]> * Adding support to register settings dynamically (#5495) * Adding support to register settings dynamically Signed-off-by: Ryan Bogan <[email protected]> * Update CHANGELOG Signed-off-by: Ryan Bogan <[email protected]> * Removed unnecessary registerSetting methods Signed-off-by: Ryan Bogan <[email protected]> * Change setting registration order Signed-off-by: Ryan Bogan <[email protected]> * Add unregisterSettings method Signed-off-by: Ryan Bogan <[email protected]> * Remove unnecessary feature flag Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> * Updated 1.3.7 release notes date (#5536) Signed-off-by: owaiskazi19 <[email protected]> Signed-off-by: owaiskazi19 <[email protected]> * Pre conditions check before updating weighted routing metadata (#4955) * Pre conditions check to allow weight updates for non decommissioned attribute Signed-off-by: Rishab Nahata <[email protected]> * Atomically update cluster state with decommission status and corresponding action (#5093) * Atomically update the cluster state with decommission status and its corresponding action in the same execute call Signed-off-by: Rishab Nahata <[email protected]> * Update Netty to 4.1.86.Final (#5529) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> * Update release date in 2.4.1 release notes (#5549) Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Suraj Singh <[email protected]> * Update 2.4.1 release notes (#5552) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Andriy Redko <[email protected]> * Refactor fuzziness interface on query builders (#5433) * Refactor Object to Fuzziness type for all query builders Signed-off-by: noCharger <[email protected]> * Revise on bwc Signed-off-by: noCharger <[email protected]> * Update change log Signed-off-by: noCharger <[email protected]> Signed-off-by: noCharger <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> * Upgrade lucene version (#5570) * Added bwc version 2.4.2 Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> * Added 2.4.2. Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> * Update Lucene snapshot to 9.5.0-snapshot-d5cef1c Signed-off-by: Suraj Singh <[email protected]> * Update changelog entry Signed-off-by: Suraj Singh <[email protected]> * Add 2.4.2 bwc version Signed-off-by: Suraj Singh <[email protected]> * Internal changes post lucene upgrade Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> * Add CI bundle pattern to distribution download (#5348) * Add CI bundle pattern for ivy repo Signed-off-by: Zelin Hao <[email protected]> * Gradle update Signed-off-by: Zelin Hao <[email protected]> * Extract path Signed-off-by: Zelin Hao <[email protected]> * Change with customDistributionDownloadType Signed-off-by: Zelin Hao <[email protected]> * Add default for exception handle Signed-off-by: Zelin Hao <[email protected]> * Add documentations Signed-off-by: Zelin Hao <[email protected]> Signed-off-by: Zelin Hao <[email protected]> * Bump protobuf-java from 3.21.9 to 3.21.11 in /plugins/repository-hdfs (#5519) * Bump protobuf-java from 3.21.9 to 3.21.11 in /plugins/repository-hdfs Bumps [protobuf-java](https:/protocolbuffers/protobuf) from 3.21.9 to 3.21.11. - [Release notes](https:/protocolbuffers/protobuf/releases) - [Changelog](https:/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.21.9...v3.21.11) --- updated-dependencies: - dependency-name: com.google.protobuf:protobuf-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Updated changelog Signed-off-by: Owais Kazi <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Owais Kazi <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Owais Kazi <[email protected]> Co-authored-by: Suraj Singh <[email protected]> Signed-off-by: Rishikesh1159 <[email protected]> Signed-off-by: Dhwanil Patel <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Nicholas Walter Knize <[email protected]> Signed-off-by: Ralph Ursprung <[email protected]> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Ryan Bogan <[email protected]> Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Xue Zhou <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: owaiskazi19 <[email protected]> Signed-off-by: Rishab Nahata <[email protected]> Signed-off-by: Suraj Singh <[email protected]> Signed-off-by: noCharger <[email protected]> Signed-off-by: Zelin Hao <[email protected]> Signed-off-by: Owais Kazi <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> Co-authored-by: Dhwanil Patel <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: Ashish <[email protected]> Co-authored-by: Nick Knize <[email protected]> Co-authored-by: Ralph Ursprung <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: Ryan Bogan <[email protected]> Co-authored-by: Poojita Raj <[email protected]> Co-authored-by: Xue Zhou <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Owais Kazi <[email protected]> Co-authored-by: Rishab Nahata <[email protected]> Co-authored-by: Suraj Singh <[email protected]> Co-authored-by: Louis Chu <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Zelin Hao <[email protected]> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

All conditional check on assertNull to fix flaky tests.

5812e6c

Signed-off-by: Rishikesh1159 <[email protected]>

Rishikesh1159 requested review from a team and reta as code owners December 2, 2022 23:57

Remove sys log statements.

46dd09a

Signed-off-by: Rishikesh1159 <[email protected]>

Rishikesh1159 added the skip-changelog label Dec 3, 2022

Rishikesh1159 mentioned this pull request Dec 3, 2022

[BUG] Fix flaky test org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.testReplicaThreadedUpdateToShardLimitsAndRejections #4212

Closed

apply spotless.

ae55b27

Signed-off-by: Rishikesh1159 <[email protected]>

andrross reviewed Dec 3, 2022

View reviewed changes

Add comments for the optional null check.

bdcefdb

Signed-off-by: Rishikesh1159 <[email protected]>

andrross approved these changes Dec 5, 2022

View reviewed changes

andrross merged commit 0bd3141 into opensearch-project:main Dec 5, 2022

andrross added the backport 2.x Backport to 2.x branch label Dec 5, 2022

opensearch-trigger-bot bot mentioned this pull request Dec 5, 2022

[Backport 2.x] Fix Flaky Tests testReplicaThreadedUpdateToShardLimitsAndRejections and testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections #5450

Merged

Rishikesh1159 mentioned this pull request Dec 6, 2022

[BUG] testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections is flaky #5157

Closed

ryanbogan pushed a commit that referenced this pull request Dec 13, 2022

Fix flaky ShardIndexingPressureConcurrentExecutionTests (#5439)

3776b4e

Add conditional check on assertNull to fix flaky tests. Signed-off-by: Rishikesh1159 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Flaky Tests testReplicaThreadedUpdateToShardLimitsAndRejections and testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections #5439

Fix Flaky Tests testReplicaThreadedUpdateToShardLimitsAndRejections and testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections #5439

Rishikesh1159 commented Dec 2, 2022

github-actions bot commented Dec 3, 2022

github-actions bot commented Dec 3, 2022

andrross Dec 3, 2022

Rishikesh1159 Dec 3, 2022

github-actions bot commented Dec 3, 2022

codecov-commenter commented Dec 3, 2022

github-actions bot commented Dec 3, 2022

Fix Flaky Tests testReplicaThreadedUpdateToShardLimitsAndRejections and testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections #5439

Fix Flaky Tests testReplicaThreadedUpdateToShardLimitsAndRejections and testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections #5439

Conversation

Rishikesh1159 commented Dec 2, 2022

Description

Issues Resolved

Check List

github-actions bot commented Dec 3, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Dec 3, 2022

Gradle Check (Jenkins) Run Completed with:

andrross Dec 3, 2022

Choose a reason for hiding this comment

Rishikesh1159 Dec 3, 2022

Choose a reason for hiding this comment

github-actions bot commented Dec 3, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Dec 3, 2022

Codecov Report

github-actions bot commented Dec 3, 2022

Gradle Check (Jenkins) Run Completed with: