[Backport 2.x] Fix flaky SegmentReplicationITs. #6053

opensearch-trigger-bot · 2023-01-27T22:49:35Z

Backport ade01ec from #6015.

* Fix flaky SegmentReplicationITs. This change fixes flakiness with segment replication ITs. It does this by updating the wait condition used to ensure replicas are up to date to wait until a searched docCount is reached instead of output of the Segments API that can change if there are concurrent refreshes. It also does this by updating the method used to assert segment stats to wait until the assertion holds true rather than at a point in time. This method is also updated to assert store metadata directly over API output. Signed-off-by: Marc Handalian <[email protected]> * Fix error message to print expected and actual doc counts. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> * spotless. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit ade01ec) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

codecov-commenter · 2023-01-27T23:10:28Z

Codecov Report

Merging #6053 (125aaf4) into 2.x (125aaf4) will not change coverage.
The diff coverage is n/a.

❗ Current head 125aaf4 differs from pull request most recent head f8e6a2f. Consider uploading reports for the commit f8e6a2f to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff            @@
##                2.x    #6053   +/-   ##
=========================================
  Coverage     70.54%   70.54%           
  Complexity    58994    58994           
=========================================
  Files          4769     4769           
  Lines        282621   282621           
  Branches      41167    41167           
=========================================
  Hits         199385   199385           
  Misses        66544    66544           
  Partials      16692    16692

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-01-27T23:15:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10087/
CommitID: f8e6a2f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-01-27T23:24:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/10087/

CommitID: f8e6a2f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Failed due to SegRepRelocationIT testPrimaryRelocationWithSegRepFailure, this will be fixed in backport of #5898

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure" -Dtests.seed=B13F4FEF87CFED51 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=nb -Dtests.timezone=Chile/Continental -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationRelocationIT > testPrimaryRelocationWithSegRepFailure FAILED
    java.lang.AssertionError: Count is 12 hits but 24 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([B13F4FEF87CFED51:35A0176FCD77DB7C]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:306)
        at org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure(SegmentReplicationRelocationIT.java:194)

github-actions · 2023-01-27T23:53:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/10094/
CommitID: f8e6a2f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-01-27T23:59:01Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/10094/

CommitID: f8e6a2f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Failing again with same error. This seems to be failing persistently. This may get resolved with #5898

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation" -Dtests.seed=79E996A71886126 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-Hant-TW -Dtests.timezone=Europe/Tirane -Druntime.java=17

org.opensearch.indices.replication.SegmentReplicationRelocationIT > testPrimaryRelocation FAILED
    java.lang.AssertionError: Count is 48 hits but 96 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([79E996A71886126:BF96AEB852E132C8]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:306)
        at org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation(SegmentReplicationRelocationIT.java:118)

dreamer-89 · 2023-01-28T04:20:13Z

Closing in favour of #6055

dreamer-89 approved these changes Jan 27, 2023

View reviewed changes

dreamer-89 added the skip-changelog label Jan 27, 2023

dreamer-89 closed this Jan 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] Fix flaky SegmentReplicationITs. #6053

[Backport 2.x] Fix flaky SegmentReplicationITs. #6053

opensearch-trigger-bot bot commented Jan 27, 2023

codecov-commenter commented Jan 27, 2023

github-actions bot commented Jan 27, 2023

dreamer-89 commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

dreamer-89 commented Jan 27, 2023 •

edited

Loading

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 28, 2023

[Backport 2.x] Fix flaky SegmentReplicationITs. #6053

[Backport 2.x] Fix flaky SegmentReplicationITs. #6053

Conversation

opensearch-trigger-bot bot commented Jan 27, 2023

codecov-commenter commented Jan 27, 2023

Codecov Report

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jan 27, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 27, 2023 • edited Loading

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Jan 28, 2023

dreamer-89 commented Jan 27, 2023 •

edited

Loading