IT Tests: FIM - Analysis and fix master branch failures

[Deblintrake09]

Wazuh QA: Branch	Wazuh QA: Commit	Wazuh: Tag	Wazuh: Commit
Master	eb32b09	v4.3.0	c7fb1e8

During research for Issue #2305 it was found that FIM was failing on Windows the master branch.
On further research on the Issue, it failed on other systems as well.
On this Issue we will investigate on what Systems and configurations the following modules are failing:

• FIM
• Active Response
• Github

Results from previous Research

package	Type	Modes	Results	Date	By	Notes
4.3.0-master17	Windows - Jenkins	whodata +realtime	🟢	2021/12/28	@Deblintrake09
4.3.0-master17	Windows - Local	scheduled + whodata +realtime	🔴	2021/12/28	@Deblintrake09
4.3.0-master17	Windows - Local	whodata +realtime	🔴	2021/12/28	@Deblintrake09
4.3.0-master17	Windows - Local	scheduled + whodata +realtime	🔴	2021/12/28	@Deblintrake09
4.3.0-master17	Linux - Local	scheduled + whodata +realtime	🔴	2021/12/28	@Deblintrake09

Test Results

Name	Local						Jenkins
OS	Linux		Windows	Solaris	macOS		Linux		Windows	Solaris	macOS
Target	Manager	Agent	Agent	Agent	Agent		Manager	Agent	Agent	Agent	Agent
active_response	⚫	🟢	⚫	NA	NA		🟢	🟢	🟢	NA	NA
FIM	🔴	🔵	⚫	⚫	⚫		🔴	🔴	🟢	BLOCKED	🔴
Github	⚫	⚫	⚫	⚫	⚫		⚫	⚫	⚫	⚫	⚫

NOTES

• Local Manager FIM Failure was Machine got out of disk space on test_synchronize_integritywin32.py:test_events_while_integrity_scan. No result files generated.
• Windows Jenkins is being run only on realtime and whodata modes
• Solaris fails to start running tests. It is being worked on by CICD team.

UPDATE 2022/01/04

• Relaunched Linux Test on Jenkins - Test Seemed to fail because date function was not considering changing the year when adding days. Month 12 changed to month 13.
• Package used : 4.3.0-1 (https://packages-dev.wazuh.com/pre-release)
• Test launched with parameters --tier 0 --tier 1 --tier 2 --fim_mode="whodata" --fim_mode="realtime" --fim_mode="scheduled"

Jenkins Instances

Type	Modes	Results	Date	By	Notes
Linux - Jenkins	scheduled + whodata +realtime	🟢	2022/01/04	@Deblintrake09
Linux - Jenkins	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09
Macos - Jenkins	scheduled + whodata +realtime	🟢	2022/01/04	@Deblintrake09
Macos - Jenkins	scheduled + whodata +realtime	🟢	2022/01/04	@Deblintrake09
Macos - Jenkins	scheduled + whodata +realtime	🟢	2022/01/04	@Deblintrake09
Manager - Jenkins	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09
Manager - Jenkins	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09
Windows - Jenkins	scheduled + whodata +realtime	🔴	2022/01/05	@Deblintrake09
Windows - Jenkins	whodata +realtime	🟢	2022/01/05	@Deblintrake09
Windows - Jenkins	whodata +realtime	🟢	2022/01/05	@Deblintrake09

###Local Tests

Type	Modes	Results	Date	By	Notes
Manager- Local	scheduled + whodata +realtime	🟢	2022/01/04	@Deblintrake09
Manager- Local	scheduled + whodata +realtime	🟢	2022/01/04	@Deblintrake09
Manager- Local	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09
Windows - Local	scheduled + whodata +realtime	🔴	2022/01/04	@Deblintrake09
Windows - Local	scheduled + whodata +realtime	🔴	2022/01/06	@Deblintrake09	crashed on scan_day_and_time.py
Windows - Local	scheduled + whodata +realtime	🔴	2022/01/07	@Deblintrake09
Linux - Local	scheduled + whodata +realtime	🔴	2022/01/05	@Deblintrake09	Huge amount of fails and Errors
Linux - Local	scheduled + whodata +realtime	🔴	2022/01/05	@Deblintrake09	1 Fail - 83 Errors caused by a file left behind by a test.
Linux - Local	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09	Ran with a dirty environment after the last 🔴 only deleted the file that was left behind.
Linux - Local	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09
Linux - Local	scheduled + whodata +realtime	🟢	2022/01/05	@Deblintrake09

[damarisg]

UPDATE 2022/01/19

Type	Results	Date	By	Notes
Windows - Local	🟢	2022/01/18	@Deblintrake09	whodata +realtime mode
Windows - Local	🟢	2022/01/18	@Deblintrake09	whodata +realtime mode
Windows - Local	🟢	2022/01/18	@Deblintrake09	whodata +realtime mode
Linux - Local	🟢	2022/01/18	@Deblintrake09	whodata +realtime mode
Linux - Local	🟢	2022/01/18	@Deblintrake09	whodata +realtime mode
Linux - Local	🟢	2022/01/18	@Deblintrake09	whodata +realtime mode

UPDATE 2022/01/17

Test Name	Type Error	Research	Status
test_fim\test_registry\test_registry_file_limit\test_registry_limit_values.py	Fail	Couldn't delete the database	Message that test is looking for does not exist in windows - known wazuh bug - Issue #11162
test_fim\test_registry\test_registry_basic_usage\test_basic_usage_registry_changes.py	Erratic	Solo 🟢 x2 🔴 x1 - Full registry_basic_usage folder: 🟢 x2	Waiting for Core's response
test_fim\test_registry\test_registry_checks\test_registry_check_others.py	Passed	Solo 🟢 x3 - Full test_registry_checks folder: 🟢 x3
test_fim\test_files\test_inotify\test_num_watches.py	Erratic	Solo 🟢 x2 🔴 x1 - Full test_inotify folder: 🟢
test_fim\test_files\test_max_eps\test_max_eps_synchronization.py	Passed	Only specific test 🟢 x3 - Full test_files folder: 🟢 x3 the specific test, but had failures others tests.

Skipped test_fim\test_registry\test_registry_file_limit\test_registry_limit_values.py because of Wazuh Issue #11162

UPDATE 2022/01/12

Description

After several executions on the Windows agent and Linux manager, we got a list of results.
Also, it should be noted that FIM must be run with levels 0 and 1 because there are tests with level 2 that are required refactor .
I add the list to investigate and check for faults and possible causes. They will be listed in order of priority.

Research: (10/)

• test_fim\test_files\test_basic_usage\test_basic_usage_create_scheduled.py
• test_fim\test_files\test_max_eps\test_max_eps_synchronization.py
• test_fim\test_registry\test_registry_basic_usage\test_basic_usage_registry_changes.py
• test_fim\test_registry\test_registry_checks\test_registry_check_others.py
• test_fim\test_registry\test_registry_file_limit\test_registry_limit_values.py
• test_fim\test_files\test_basic_usage\test_basic_usage_deferred_delete_folder.py
• test_fim\test_files\test_inotify\test_num_watches.py
• test_fim\test_files\test_max_files_per_second\test_max_files_per_second.py
• test_fim\test_files\test_report_changes\test_disk_quota_disabled.py
• tests\integration\test_fim\test_files\test_file_limit\test_file_limit_delete_full.py

Failed: (8/)

• test_fim/test_files/test_audit/test_remove_rule_five_times.py
• test_fim/test_files/test_audit/test_audit.py
• test_fim/test_files/test_audit/test_audit_after_initial_scan.py
• test_fim/test_files/test_wildcards_complex/test_wildcards_complex.py >> It should be skipped by Roadmap of refactor and test blocked #2174
• test_fim/test_files/test_wildcards_complex/test_wildcards_complex_runtime.py >> It should be skipped by Roadmap of refactor and test blocked #2174
• tests\integration\test_fim\test_files\test_basic_usage\test_basic_usage_baseline_generation.py >> Associate to Roadmap of refactor and test blocked #2174
• tests\integration\test_fim\test_files\test_max_eps\test_max_eps_synchronization.py
• tests\integration\test_fim\test_synchronization\test_synchronize_integrity_win32.py
• test_fim\test_registry\test_registry_ambiguous_confs\test_registry_ambiguous_duplicated_entries.py
• test_fim\test_registry\test_registry_ambiguous_confs\test_registry_ambiguous_simple.py
• test_fim\test_registry\test_registry_nodiff\test_registry_no_diff.py
• test_fim\test_registry\test_registry_report_changes\test_registry_report_changes.py

- Errors: (24/)

• test_fim/test_files/test_audit/test_audit_no_dir.py::test_audit_no_dir[get_configuration0-tags_to_apply0] x1
• test_fim\test_files\test_checks\test_check_all.py >> It should be skipped by Refactor FileMonitor because of inconsistent behavior when heavy log activity #1602
• test_fim\test_files\test_file_limit\test_file_limit_capacity_alerts.py
• test_fim\test_files\test_max_eps\test_max_eps.py
• test_fim/test_files/test_follow_symbolic_link/test_audit_rules_removed_after_change_link.py
• test_fim/test_files/test_follow_symbolic_link/test_change_target.py
• test_fim/test_files/test_follow_symbolic_link/test_change_target_inside_folder.py
• test_fim/test_files/test_follow_symbolic_link/test_change_target_with_nested_directory.py
• test_fim/test_files/test_follow_symbolic_link/test_delete_symlink.py
• test_fim/test_files/test_follow_symbolic_link/test_delete_target.py
• test_fim/test_files/test_follow_symbolic_link/test_follow_symbolic_disabled.py
• test_fim/test_files/test_follow_symbolic_link/test_monitor_symlink.py
• test_fim/test_files/test_follow_symbolic_link/test_not_following_symbolic_link.py
• test_fim/test_files/test_follow_symbolic_link/test_revert_symlink.py
• tests\integration\test_fim\test_files\test_basic_usage\test_basic_usage_deferred_delete_folder.py
• tests\integration\test_fim\test_files\test_checks\test_check_all.py
• tests\integration\test_fim\test_files\test_inotify\test_num_watches.py
• tests\integration\test_fim\test_files\test_max_files_per_second\test_max_files_per_second.py
• tests\integration\test_fim\test_files\test_report_changes\test_file_size_default.py >> It should be skipped by Refactor FileMonitor because of inconsistent behavior when heavy log activity #1602
• tests\integration\test_fim\test_files\test_report_changes\test_file_size_disabled.py
• tests\integration\test_fim\test_synchronization\test_sync_disabled_win32.py
• tests\integration\test_fim\test_synchronization\test_sync_enabled_win32.py
• tests\integration\test_fim\test_synchronization\test_sync_registry_disabled_win32.py
• tests\integration\test_fim\test_synchronization\test_sync_registry_enabled_win32.py
• test_fim\test_files\test_ambiguous_confs\test_ignore_works_over_restrict.py