Add script to parse and obtain stats from cluster CSVs #2032
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Selutario
force-pushed
the
feature/1938-cluster-stas-script
branch
from
2d9cdae
to
7275730
Compare
AdriiiPRodri
suggested changes
Oct 27, 2021
deps/wazuh_testing/wazuh_testing/tools/performance/csv_parser.py
Outdated
Show resolved
Hide resolved
davidjiglesias
approved these changes
Oct 27, 2021
Selutario
added
team/framework
test/performance
and removed
team/framework
test/performance
labels
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new tool inside
wazuh_testingpackage that can be used to load data from CSVs and calculate some stats from it.There are two similar tools:
The objective of both tools is to parse the CSVs of all the nodes in a cluster environment, calculate statistics for each of them (mean and regression coefficient), select and return the highest values:
ClusterCSVTasksParseris used to obtain this information from the cluster tasks (agent-info sync, Integrity check and Integrity Sync), whileClusterCSVResourcesParsercan be used to obtain information of the resource-usage of the wazuh-clusterd process (RAM, CPU, File descriptors by default).Configuration options
artifacts_path (
ClusterCSVTasksParser,ClusterCSVResourcesParser): Specifies the path where the CSVs of the cluster nodes are located. It must follow this format:columns (
ClusterCSVResourcesParser): Columns of the CSV on which statistics should be extracted.Output example
TASKS
{ "setup_phase": { "integrity_check": { "time_spent(s)": { "workers": { "mean": ("worker_4", 2.882842105263158), "max": ("worker_10", 12.238), }, "master": { "mean": ("master", 1.7251713780918725), "max": ("master", 10.025), }, } }, "integrity_sync": { "time_spent(s)": { "workers": { "mean": ("worker_3", 0.8801250000000002), "max": ("worker_7", 6.87), }, "master": { "mean": ("master", 3.7857918149466196), "max": ("master", 23.085), }, } }, "agent-info_sync": { "time_spent(s)": { "workers": { "mean": ("worker_4", 2.0509978833333333), "max": ("worker_3", 9.55), }, "master": { "mean": ("master", 0.4447335640138408), "max": ("master", 3.676), }, } }, }, "stable_phase": { "integrity_check": { "time_spent(s)": { "workers": { "mean": ("worker_8", 1.6634466019417473), "max": ("worker_8", 3.108), }, "master": { "mean": ("master", 0.726184287099903), "max": ("master", 2.026), }, } }, "agent-info_sync": { "time_spent(s)": { "workers": { "mean": ("worker_8", 0.7170288461538461), "max": ("worker_8", 1.881), }, "master": { "mean": ("master", 0.2620988372093024), "max": ("master", 1.107), }, } }, }, }RESOURCES
{ "setup_phase": { "wazuh-clusterd": { "USS(KB)": { "workers": { "mean": ("worker_9", 75079.43661971831), "max": ("worker_10", 149044.0), "reg_cof": ("worker_9", 489.7276436479312), }, "master": { "mean": ("master", 163284.44755244756), "max": ("master", 347204.0), "reg_cof": ("master", 1759.8759644111753), }, }, "CPU(%)": { "workers": { "mean": ("worker_8", 8.525352112676055), "max": ("worker_10", 38.3), "reg_cof": ("worker_5", 0.04679342234032074), }, "master": { "mean": ("master", 52.511888111888105), "max": ("master", 85.1), "reg_cof": ("master", 0.28662464296267104), }, }, "FD": { "workers": { "mean": ("worker_6", 14.195804195804195), "max": ("worker_5", 16), "reg_cof": ("worker_3", 0.0024418070192718387), }, "master": { "mean": ("master", 42.61538461538461), "max": ("master", 86), "reg_cof": ("master", -0.03370924849798087), }, }, } }, "stable_phase": { "wazuh-clusterd": { "USS(KB)": { "workers": { "mean": ("worker_9", 106824.18433179724), "max": ("worker_7", 172632.0), "reg_cof": ("worker_1", 9.019405572231445), }, "master": { "mean": ("master", 218785.16129032258), "max": ("master", 281268.0), "reg_cof": ("master", -70.75089840612243), }, }, "CPU(%)": { "workers": { "mean": ("worker_8", 10.96589861751152), "max": ("worker_2", 25.8), "reg_cof": ("worker_4", 0.0013146089056121396), }, "master": { "mean": ("master", 41.93041474654378), "max": ("master", 61.3), "reg_cof": ("master", -0.002096750705806691), }, }, "FD": { "workers": { "mean": ("worker_5", 14.055299539170507), "max": ("worker_4", 15), "reg_cof": ("worker_6", 0.0001952869169673985), }, "master": { "mean": ("master", 24.341013824884794), "max": ("master", 29), "reg_cof": ("master", -0.014530691432141685), }, }, } }, }