Feature(TopologyOpsMonitor): Monitor decommission topology operation #8843
+131
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aleksbykov
added
Ready for review
backport/2024.2
aleksbykov
force-pushed
the
fix-issue-8144
branch
from
bf17a6f
to
b0788d4
Compare
aleksbykov
force-pushed
the
fix-issue-8144
branch
from
b0788d4
to
b3f5535
Compare
Decommission operation could be terminated for different reason, but decommission process will be run on node and could successfully be decommissioned. But because of nemesis is terminated by exception cluster health validator could abort whole test run because node status is Decommissioning. This start happened with nemesis DecommissionStreamingErr, when log message could not be found and process is aborted by timeout, while decommission continue to run. To Catch such case FailedDecommissionOperationMonitoring is presented It is ContextManager which could be used to safely run decommission operation and check node status and wait decommission will be finished if command aborted or terminated. Fix: scylladb#8144
aleksbykov
force-pushed
the
fix-issue-8144
branch
from
b3f5535
to
0caeb17
Compare
|
@soyacz , can you take a look? |
soyacz
reviewed
Oct 7, 2024
| self.target_node = target_node | ||
| self.db_cluster: "BaseScyllaCluster" = target_node.parent_cluster | ||
| self.target_node_ip = target_node.ip_address | ||
| expected_exception = expected_exception or set() |
There was a problem hiding this comment.
do we need that? From my understanding it is used to skip waiting for finish node decommission if test ended (KillNemesis exception raised).
At least rename it to something that reflects that (FailedDecommissionOperationMonitoring that expects KillNemesis is weird)
| LOGGER.warning("Decommission failed with error: %s", traceback.format_exception(exc_type, exc_val, exc_tb)) | ||
| decommission_in_progress = self.is_node_decommissioning() | ||
| if not decommission_in_progress: | ||
| self.db_cluster.verify_decommission(self.target_node) |
There was a problem hiding this comment.
verify_decommission terminates the node and does bunch of sttuff - how long it takes? Maybe we could also skip that in case of KillNemesis exception?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Decommission operation could be terminated for different reason, but decommission process will be run on node and could successfully be decommissioned. But because of nemesis is terminated by exception cluster health validator could abort whole test run because node status is Decommissioning. This start happened with nemesis DecommissionStreamingErr, when log message could not be found and process is aborted by timeout, while decommission continue to run.
To Catch such case FailedDecommissionOperationMonitoring is presented It is ContextManager which could be used to safely run decommission operation and check node status and wait decommission will be finished if command aborted or terminated.
Fix: #8144
Testing
PR pre-checks (self review)
backportlabelsReminders
sdcm/sct_config.py)unit-test/folder)