-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Backport to 2.x] [Segment Replication] - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards. #4450
Conversation
Looks like something is missing. There is only one file change in this backport PR. |
469228a
to
62049fd
Compare
Gradle Check (Jenkins) Run Completed with:
|
Java compilation error :(
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Java spotless failure. |
Gradle Check (Jenkins) Run Completed with:
|
|
…uting metadata snapshot on primary shards. (opensearch-project#4366) * Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards. This change fixes the errors that occur when computing metadata snapshots on primary shards from the latest in-memory SegmentInfos. The error occurs when a segments_N file that is referenced by the in-memory infos is deleted as part of a concurrent commit. The segments themselves are incref'd by IndexWriter.incRefDeleter but the commit file (Segments_N) is not. This change resolves this by ignoring the segments_N file when computing metadata for CopyState and only sending incref'd segment files to replicas. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> * Update StoreTests.testCleanupAndPreserveLatestCommitPoint to assert additional segments are deleted. Signed-off-by: Marc Handalian <[email protected]> * Rename snapshot to metadataMap in CheckpointInfoResponse. Signed-off-by: Marc Handalian <[email protected]> * Refactor segmentReplicationDiff method to compute off two maps instead of MetadataSnapshots. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless. Signed-off-by: Marc Handalian <[email protected]> * Revert catchall in SegmentReplicationSourceService. Signed-off-by: Marc Handalian <[email protected]> * Revert log lvl change. Signed-off-by: Marc Handalian <[email protected]> * Fix SegmentReplicationTargetTests Signed-off-by: Marc Handalian <[email protected]> * Cleanup unused logger. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]> Co-authored-by: Suraj Singh <[email protected]>
…d of relying on segments_N from primary shards. (opensearch-project#4402) * Segment Replication - Update replicas to commit SegmentInfos instead of relying on segments_N from primary shards. This change updates replicas to commit SegmentInfos before the shard is closed, on receiving a new commit point from a primary, and when a new primary is detected. This change also makes the public commitSegmentInfos on NRTEngine obsolete, refactoring IndexShard to simply call reset on the engine. Signed-off-by: Marc Handalian <[email protected]> * Remove noise & extra log statement. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: Marc Handalian <[email protected]>
…anslog() Signed-off-by: Suraj Singh <[email protected]>
Signed-off-by: Suraj Singh <[email protected]>
407dc92
to
8a18787
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the backport @dreamer-89
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## 2.x #4450 +/- ##
============================================
- Coverage 70.67% 70.62% -0.05%
+ Complexity 57179 57153 -26
============================================
Files 4585 4585
Lines 274506 274507 +1
Branches 40227 40229 +2
============================================
- Hits 193996 193878 -118
- Misses 64278 64389 +111
- Partials 16232 16240 +8
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Manual backport of #4402 to 2.x