Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport to 2.x] [Segment Replication] - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards. #4450

Merged
merged 4 commits into from
Sep 7, 2022
Merged

[Backport to 2.x] [Segment Replication] - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards. #4450

merged 4 commits into from
Sep 7, 2022

Commits on Sep 7, 2022

  1. Segment Replication - Fix NoSuchFileException errors caused when comp… 

    …uting metadata snapshot on primary shards. (opensearch-project#4366)

* Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards.

This change fixes the errors that occur when computing metadata snapshots on primary shards from the latest in-memory SegmentInfos.  The error occurs when a segments_N file that is referenced by the in-memory infos is deleted as part of a concurrent commit.  The segments themselves are incref'd by IndexWriter.incRefDeleter but the commit file (Segments_N) is not.  This change resolves this by ignoring the segments_N file when computing metadata for CopyState and only sending incref'd segment files to replicas.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless.

Signed-off-by: Marc Handalian <[email protected]>

* Update StoreTests.testCleanupAndPreserveLatestCommitPoint to assert additional segments are deleted.

Signed-off-by: Marc Handalian <[email protected]>

* Rename snapshot to metadataMap in CheckpointInfoResponse.

Signed-off-by: Marc Handalian <[email protected]>

* Refactor segmentReplicationDiff method to compute off two maps instead of MetadataSnapshots.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless.

Signed-off-by: Marc Handalian <[email protected]>

* Revert catchall in SegmentReplicationSourceService.

Signed-off-by: Marc Handalian <[email protected]>

* Revert log lvl change.

Signed-off-by: Marc Handalian <[email protected]>

* Fix SegmentReplicationTargetTests

Signed-off-by: Marc Handalian <[email protected]>

* Cleanup unused logger.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: Suraj Singh <[email protected]>
    @mch2 @dreamer-89
    mch2 and dreamer-89 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    0b85fd7 View commit details
    Browse the repository at this point in the history

  2. [Segment Replication] - Update replicas to commit SegmentInfos instea… 

    …d of relying on segments_N from primary shards. (opensearch-project#4402)

* Segment Replication - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards.

This change updates replicas to commit SegmentInfos before the shard is closed, on receiving a new commit point from a primary, and when a new primary is detected. This change also makes the public commitSegmentInfos on NRTEngine obsolete, refactoring IndexShard to simply call reset on the engine.

Signed-off-by: Marc Handalian <[email protected]>

* Remove noise & extra log statement.

Signed-off-by: Marc Handalian <[email protected]>

* PR feedback.

Signed-off-by: Marc Handalian <[email protected]>

Signed-off-by: Marc Handalian <[email protected]>
    @mch2 @dreamer-89
    mch2 authored and dreamer-89 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    e1a3a9f View commit details
    Browse the repository at this point in the history

  3. [Segment Replicaiton] Fix merge conflict. Update EngineTestCase.getTr… 

    …anslog()

Signed-off-by: Suraj Singh <[email protected]>
    @dreamer-89
    dreamer-89 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    bb6db6d View commit details
    Browse the repository at this point in the history

  4. Spotless check apply fixes 

    Signed-off-by: Suraj Singh <[email protected]>
    @dreamer-89
    dreamer-89 committed Sep 7, 2022
    Configuration menu
    Copy the full SHA
    8a18787 View commit details
    Browse the repository at this point in the history