Handle data corruption error in replication #3895
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yycptt
reviewed
Jan 14, 2021
| @@ -176,11 +176,11 @@ func (h *nosqlHistoryManager) ReadHistoryBranch( | |||
|
|
|||
| switch { | |||
| case nodeID < lastNodeID: | |||
| return nil, &types.InternalServiceError{ | |||
| return nil, &types.InternalDataInconsistencyError{ | |||
There was a problem hiding this comment.
There're some more checks in historyStore.go regarding corrupted history. Can we also return this inconsistency error if those checks fail?
There was a problem hiding this comment.
Updated those errors in the history store. I think there might be more cases to be converted. I don't plan to convert all of them in this change. We can continue with other changes.
There was a problem hiding this comment.
Sure thing. I mentioned those in historyStore because I saw them a lot in production logs. We can gradually convert the rest.
yycptt
approved these changes
Jan 14, 2021
github-actions
bot
force-pushed
the
data_corruption
branch
from
c4b6965
to
b05b60e
Compare
yux0
added a commit
that referenced
this pull request
Jan 14, 2021
github-actions bot
pushed a commit
to vytautas-karpavicius/cadence
that referenced
this pull request
Feb 4, 2021
yux0
added a commit
to yux0/cadence
that referenced
this pull request
May 4, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Update some internal service error to internal data inconsistency error and handle data corruption error in replication.
Why?
Those data inconsistency error could block the replication stack.
How did you test it?
Added unit tests.
Potential risks