From 87a6bde0d391f1b483d0edf8d4615afbf5869f6e Mon Sep 17 00:00:00 2001 From: Nico Stefani Date: Fri, 28 Apr 2023 11:46:02 -0300 Subject: [PATCH] feat(#3333): add AWS integration tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(#3333): Add basic structure for AWS tests * feat(#3335): Cloudtrail integration tests (#3624) * feat(#3335): add basic test for aws cloudtrail module * feat(#3335): add utils for handle files in S3 buckets * feat(#3335): add remove_from_bucket test for aws cloudtrail module * feat(#3335): add fixture for delete s3_cloudtrail.db * refactor(#3335): rename modules * refactor(#3335): change the return of upload_file function * refactor(#3335): Fix docstring and rename constant * refactor(#3335): improve default test catching parser and module errors * feat(#3335): add only_logs_from test for aws cloudtrail module * feat(#3335): add path tests for aws cloudtrail module * feat(#3335): add path_suffix tests for aws cloudtrail module * feat(#3335): add regions tests for aws cloudtrail module * refactor(#3335): Move wazuh_log_monitor to a fixture * feat(#3335): add discard_regex tests for aws cloudtrail module * feat(#3335): add only_logs_after tier_1 test for aws cloudtrail module * refactor(#3335): Improve name of test cases * refactor(#3335): Improve tests assertions * refactor(#3335): Improve tier 1 test * refactor(#3335): Implement new design for Tier 1 test * style(#3335): Changed the extension from .yml to .yaml * style(#3335): Fixed linter issues * style(#3335): Apply suggestions from code review Co-authored-by: José Luis López Sánchez * refactor(#3335): Add path parameter to get_db_connection * style(#3335): Fix quotes use according to QA style * style(#3335): Use google docstring style * refactor(#3335): Add missing information about fixtures * refactor(#3335): Change sqlite table name for backward compatibility * style(#3335): Fix docstrings punctuation and capitalization * refactor(#3335): Apply suggestions from code review * style(#3335): Apply suggestions from code review Fix punctuation in docstrings. Co-authored-by: David Jose Iglesias Lopez * style(#3335): Apply suggestions from code review Co-authored-by: David Jose Iglesias Lopez Co-authored-by: José Luis López Sánchez Co-authored-by: David Jose Iglesias Lopez * feat(#3336): VPC integration tests (#3699) * feat(#3336): Add basic test for aws vpc integration * feat(#3336): Add data generator for vpcflow type * feat(#3336): Add remove_from_bucket test for aws vpc integration * feat(#3336): Add only_logs_after tests for aws vpc integration * feat(#3336): Add path tests for aws vpc integration * feat(#3336): Add path_suffix tests for aws vpc integration * feat(#3336): Add regions tests for aws vpc integration * feat(#3336): Add discard_regex tests for aws vpc integration * feat(#3336): Add only_logs_after tier_1 tests for aws vpc integration * style(#3336): Fix linter issues * feat(#3336): Use join in path builds * feat(#3336): Fix cloudtrail cases * feat(#3336): Detect found and skipped logs * feat(#3336): Improve delete_file_from_s3 fixture * fix(#3336): Add minor fixes --------- Co-authored-by: David Jose Iglesias Lopez * refactor(#3333): Use utcnow() instead of now() * feat(#3337): AWS Config integration tests (#3705) * feat(#3337): Add basic test for aws config integration * feat(#3337): Add remove_from_bucket test for aws config integration * feat(#3337): Add only_logs_after tests for aws config integration * feat(#3337): Add path tests for aws config integration * feat(#3337): Add path_suffix tests for aws config integration * feat(#3337): Add regions tests for aws config integration * feat(#3337): Add discard_regex tests for aws config integration * feat(#3337): Add only_logs_after tier_1 tests for aws config integration * feat(#3337): Use non padaded month format * feat(#3337): Use join in path builds * feat(#3337): Improve message for check_non_processed_logs_from_output * fix(#3337): Add minor fixes * feat(#3338): ALB, CLB and NLB integration tests (#3711) * feat(#3338): Add basic test for aws ALB integration * feat(#3338): Add remove_from_bucket test for aws ALB integration * feat(#3338): Add schema for ALB table * feat(#3338): Add only_logs_after tests for aws ALB integration * feat(#3338): Add path tests for aws ALB integration * feat(#3338): Add regions tests for aws ALB integration * feat(#3338): Add regions discard_regex for aws ALB integration * feat(#3338): Add regions only_logs_after tier 1 for aws ALB integration * feat(#3338): Improve upload_and_delete_file_to_s3 fixture * feat(#3338): Skip remove_from_bucket test case for ALB * feat(#3338): Add basic test for aws CLB integration * feat(#3338): Add remove_from_bucket test for aws CLB integration * feat(#3338): Add only_logs_after tests for aws CLB integration * feat(#3338): Add path tests for aws CLB integration * feat(#3338): Add regions only_logs_after tier 1 for aws CLB integration * feat(#3338): Add regions discard_regex for aws CLB integration * feat(#3338): Add only_logs_after tier 1 for aws CLB integration * feat(#3338): Skip remove_from_bucket test case for CLB * feat(#3338): Add basic test for aws NLB integration * feat(#3338): Add remove_from_bucket test for aws NLB integration * feat(#3338): Add only_logs_after tests for aws NLB integration * feat(#3338): Add path tests for aws NLB integration * feat(#3338): Add regions tests for aws NLB integration * feat(#3338): Add only_logs_after tier 1 for aws NLB integration * style(#3335): Fix linter issues * feat(#3338): Move skip function to a fixture * feat(#3338): Use join in path builds * feat(#3338): Improve docstrings * feat(#3338): Increase timeout to avoid random failures * refactor(#3338): Use utcnow() instead of now() and typos * refactor(#3338): Apply changes from CR * feat(#3339): KMS integration tests (#3715) * feat(#3339): Add basic test for aws KMS integration * feat(#3339): Add data generator for KMS type * feat(#3339): Add remove_from_bucket test for aws KMS integration * feat(#3339): Add schema for custom table * feat(#3339): Add only_logs_after tests for aws KMS integration * feat(#3339): Add path tests for aws KMS integration * feat(#3339): Add discard_regex tests for aws KMS integration * feat(#3339): Adapt only_logs_after tier 1 test to custom types * feat(#3339): Add only_logs_after tier 1 for aws KMS integration * feat(#3339): Use join in path builds * feat(#3339): Improve skipped logs count in tier 1 test * feat(#3339): Add missing docstring * style(#3339): Apply suggestions from code review Co-authored-by: Facundo Dalmau * refactor(#3339): Use utcnow() instead of now() and typos * fix(#3339): Apply suggestions from code review Co-authored-by: David Jose Iglesias Lopez --------- Co-authored-by: Facundo Dalmau Co-authored-by: David Jose Iglesias Lopez * feat(#3340): Macie integration tests (#3734) * feat(#3340): Add basic test for aws Macie integration * feat(#3336): Add data generator for vpcflow type * feat(#3340): Add remove_from_bucket test for aws Macie integration * feat(#3340): Add only_logs_after tests for aws Macie integration * feat(#3340): Add path tests for aws Macie integration * feat(#3340): Add discard_regex tests for aws Macie integration * feat(#3340): Add only_logs_after tier 1 for aws Macie integration * feat(#3340): Use join in path builds * fix(#3340): Fix typo * fix(#3340): Remove unused arguments * style(#3340): Apply suggestions from code review Co-authored-by: David Jose Iglesias Lopez --------- Co-authored-by: David Jose Iglesias Lopez * fix(#3333): Improve test condition * feat(#3341): Trusted Advisor integration tests (#3736) * feat(#3341): Add basic test for aws Trusted Advisor integration * feat(#3339): Add data generator for KMS type * feat(#3341): Add remove_from_bucket test * feat(#3341): Add only_logs_after tests for aws Trusted Advisor integration * feat(#3341): Add path tests for aws Trusted Advisor integration * feat(#3341): Add discard_regex tests for aws Trusted Advisor integration * feat(#3341): Add only_logs_after tier 1 for aws Trusted Advisor integration * feat(#3342): GuardDuty integration tests (#3762) * feat(#3342): Add basic test for aws GuardDuty integration * feat(#3342): Add remove_from_bucket test for aws Guard Duty integration * feat(#3342): Add schema for GuardDuty table * feat(#3342): Add only_logs_after tests for aws GuardDuty integration * feat(#3342): Add path tests for aws GuardDuty integration * feat(#3342): Add discard_regex tests for aws GuardDuty integration * feat(#3342): Add only_logs_after tier 1 for aws GuarDuty integration * fix(#3342): Adjust the number of found logs * feat(#3342): Add missing docstring * feat(#3342): Add data generator for Native GD * feat(#3342): Add test cases for Native GuardDuty * refactor(#3342): Improve tier 1 test findings * fix(#3333): Improve some test findings * feat(#3343): WAF integration tests (#3763) * feat(#3343): Add basic test for aws WAF integration * feat(#3343): Add data generator for WAF type * feat(#3343): Add remove_from_bucket test for aws WAF integration * feat(#3343): Add schema for WAF table * feat(#3343): Add only_logs_after tests for aws WAF integration * feat(#3343): Add path tests for aws WAF integration * feat(#3343): Add discard_regex tests for aws WAF integration * feat(#3343): Add only_logs_after tier 1 for aws WAF integration * style(#3343): Apply suggestions from code review Co-authored-by: Facundo Dalmau * refactor(#3343): Improve discard_regex test findings --------- Co-authored-by: Facundo Dalmau * feat(#3344): Server Access integration tests (#3771) * feat(#3344): Add basic test for aws server access integration * feat(#3344): Add data generator for Server Access type * feat(#3344): Add remove_from_bucket test for aws SA integration * feat(#3344): Add schema for ServerAccess table * feat(#3344): Add only_logs_after tests for aws SA integration * feat(#3344): Add path tests for aws SA integration * feat(#3344): Add discard_regex tests for aws FA integration * feat(#3344): Add only_logs_after tier 1 for aws SA integration * feat(#3344): Use table_name instead of bucket_type * style(#3344): Apply suggestions from code review Co-authored-by: Facundo Dalmau --------- Co-authored-by: Facundo Dalmau * feat(#3345): Add basic test for aws inspector integration * feat(#3345): Add only_logs_after tests for aws Inspector integration * feat(#3345): Add only_logs_after tier 1 for Inspector integration * refactor(#3345): Minor fixes * style(#3345): Apply suggestions from code review Co-authored-by: Facundo Dalmau * style(#3345): Apply suggestions from code review Co-authored-by: José Luis López Sánchez * feat(#3346): CloudWatch integration tests (#3857) * feat(#3346): Add basic test for aws cloudwatch integration * feat(#3346): Rename basic test for buckets integration * feat(#3346): Add tools for cloudwatch integration * feat(#3346): Add remove_log_streams test for aws cloudwatch integration * feat(#3346): Add utils for cloudwatch tests * feat(#3346): Add fixtures for cloudwatch tests * feat(#3346): Add only_logs_after tests for aws CloudWatch integration * feat(#3346): Rename only_logs_after tests for buckets integration * feat(#3346): Add regions tests for aws CloudWatch integration * feat(#3346): Rename regions tests for buckets integration * feat(#3346): Add log_groups tests for aws CloudWatch integration * feat(#3346): Add only_logs_after tier 1 for aws CloudWatch integration * feat(#3346): Rename tier 1 tests for buckets integration * refactor(#3346): Minor fixes * style(#3346): Apply suggestions from code review Co-authored-by: Facundo Dalmau * style(#3346): Apply suggestions from code review Co-authored-by: Facundo Dalmau --------- Co-authored-by: Facundo Dalmau * feat(#3347): Umbrella integration tests (#3796) * feat(#3347): Add basic test for aws umbrella integration * feat(#3347): Add data generator for Umbrella type * feat(#3347): Add remove_from_bucket test for aws Umbrella integration * feat(#3347): Add schema for Umbrella table * feat(#3347): Add only_logs_after tests for aws Umbrella integration * feat(#3347): Add path tests for aws Umbrella integration * feat(#3347): Add discard_regex tests for aws Umbrella integration * feat(#3347): Add only_logs_after tier 1 for aws Umbrella integration * style(#3347): Apply suggestions from code review Co-authored-by: Facundo Dalmau * fix(#3347): Fix discard_regex test findings --------- Co-authored-by: Facundo Dalmau * fix(#3333): Add missing PATH with empty value * feat(#3581): Parser integration tests (#3882) * feat(#3581): Add tests for mandatory missing params * feat(#3581): Add tests for empty values * feat(#3581): Add tests for invalid values * feat(#3581): Add tests for multiple bucket and service tags * style(#3581): Apply suggestions from code review Co-authored-by: Facundo Dalmau * style(#3581): Apply suggestions from code review Co-authored-by: Facundo Dalmau --------- Co-authored-by: Facundo Dalmau * fix(#3333): Adjust some tests values * fix(#3333): Fix cloudwatchlogs_log_groups_with_data test * fix(#3333): Adjust cisco_umbrella_only_logs_after_multiple_calls values * fix(#3333): Normalize tier1 tests for all bucket integrations * feat(#3348): AWS integration tests readme (#3892) * docs(#3348): Add AWS integration tests README * docs(#3348): Add referece to AWS tests * refactor(#3348): Improve explanation about credentials Co-authored-by: Facundo Dalmau --------- Co-authored-by: Facundo Dalmau * fix(#3333): Adjust inexistent_region test for services * fix(#3333): Adjust parser tests * fix(#3333): Change message to search for services region tests * fix(#3333): Marked inspector tier 1 test as xfail * fix(#3333): Splitted inspector and cloudwatch logs * fix(#3333): Timeouts adjustments * fix(#3333): Fixed tests cases for cloudwatch tier 1 tests * fix(#3333): Fixed linter issues * docs(#3333): Added changelog * fix(#3333): Fixed order and format * fix(#3333): Moved constants to __init__.py * fix(#3333): Replaced wazuh_log_monitor with file_monitoring * fix(#3333): Moved constants to wazuh_testing/__init__.py * fix(#3333): Moved constants to test_aws/__init__.py * fix(#3333): Improved docstring * fix(#3333): Moved exception to new file * fix(#3333): Improved fixtures and removed unused ones * fix(#3333): Fixed quotation marks * fix(#3333): Removed type hints * fix(#3339): Apply suggestions from code review Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com> * fix(#3333): Install boto3 only in linux * fix(#3339): Apply suggestions from code review Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com> * fix(#3333): Moved constants --------- Co-authored-by: José Luis López Sánchez Co-authored-by: David Jose Iglesias Lopez Co-authored-by: Facundo Dalmau Co-authored-by: Julia Magán <80041853+juliamagan@users.noreply.github.com> --- CHANGELOG.md | 1 + deps/wazuh_testing/wazuh_testing/__init__.py | 2 + .../modules/api/event_monitor.py | 3 +- .../wazuh_testing/modules/aws/__init__.py | 58 + .../wazuh_testing/modules/aws/cli_utils.py | 61 + .../modules/aws/cloudwatch_utils.py | 87 ++ .../modules/aws/data_generator.py | 1183 +++++++++++++++++ .../wazuh_testing/modules/aws/db_utils.py | 256 ++++ .../modules/aws/event_monitor.py | 329 +++++ .../wazuh_testing/modules/aws/exceptions.py | 2 + .../wazuh_testing/modules/aws/s3_utils.py | 89 ++ .../wazuh_testing/tools/utils.py | 16 +- requirements.txt | 1 + tests/integration/README.md | 5 +- tests/integration/test_aws/README.md | 180 +++ tests/integration/test_aws/conftest.py | 161 +++ .../bucket_configuration_defaults.yaml | 15 + .../cloudwatch_configuration_defaults.yaml | 17 + .../inspector_configuration_defaults.yaml | 15 + .../configuration_discard_regex.yaml | 23 + .../configuration_log_groups.yaml | 19 + ...et_configuration_with_only_logs_after.yaml | 19 + ...configuration_without_only_logs_after.yaml | 17 + ...ch_configuration_with_only_logs_after.yaml | 19 + ...or_configuration_with_only_logs_after.yaml | 17 + ...configuration_without_only_logs_after.yaml | 17 + ...figuration_bucket_and_service_missing.yaml | 7 + ...tion_multiple_bucket_and_service_tags.yaml | 47 + .../configuration_type_missing_in_bucket.yaml | 13 + ...configuration_type_missing_in_service.yaml | 13 + .../configuration_values_in_bucket.yaml | 25 + .../configuration_values_in_service.yaml | 21 + .../configuration_path_suffix.yaml | 19 + .../path_test_module/configuration_path.yaml | 19 + .../bucket_configuration_regions.yaml | 19 + .../cloudwatch_configuration_regions.yaml | 19 + .../inspector_configuration_regions.yaml | 17 + .../configuration_remove_from_bucket.yaml | 19 + .../configuration_remove_log_stream.yaml | 19 + .../cases_bucket_defaults.yaml | 125 ++ .../cases_cloudwatch_defaults.yaml | 8 + .../cases_inspector_defaults.yaml | 6 + .../cases_discard_regex.yaml | 238 ++++ .../cases_log_groups.yaml | 21 + .../cases_bucket_multiple_calls.yaml | 98 ++ .../cases_bucket_with_only_logs_after.yaml | 183 +++ .../cases_bucket_without_only_logs_after.yaml | 155 +++ .../cases_cloudwatch_multiple_calls.yaml | 6 + ...cases_cloudwatch_with_only_logs_after.yaml | 12 + .../cases_inspector_multiple_calls.yaml | 5 + .../cases_inspector_with_only_logs_after.yaml | 9 + ...cases_service_without_only_logs_after.yaml | 9 + .../cases_bucket_and_service_missing.yaml | 4 + .../cases_empty_values_in_bucket.yaml | 71 + .../cases_empty_values_in_service.yaml | 39 + .../cases_invalid_values_in_bucket.yaml | 83 ++ .../cases_invalid_values_in_service.yaml | 49 + ...ases_multiple_bucket_and_service_tags.yaml | 4 + .../cases_type_missing_in_bucket.yaml | 4 + .../cases_type_missing_in_service.yaml | 4 + .../cases_path_suffix.yaml | 116 ++ .../path_test_module/cases_path.yaml | 548 ++++++++ .../cases_bucket_regions.yaml | 233 ++++ .../cases_cloudwatch_regions.yaml | 38 + .../cases_inspector_regions.yaml | 32 + .../cases_remove_from_bucket.yaml | 140 ++ .../cases_remove_log_streams.yaml | 8 + tests/integration/test_aws/test_basic.py | 315 +++++ .../test_aws/test_discard_regex.py | 142 ++ tests/integration/test_aws/test_log_groups.py | 157 +++ .../test_aws/test_only_logs_after.py | 918 +++++++++++++ tests/integration/test_aws/test_parser.py | 597 +++++++++ tests/integration/test_aws/test_path.py | 155 +++ .../integration/test_aws/test_path_suffix.py | 157 +++ tests/integration/test_aws/test_regions.py | 443 ++++++ .../test_aws/test_remove_from_bucket.py | 225 ++++ 76 files changed, 8220 insertions(+), 6 deletions(-) create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/__init__.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/cli_utils.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/cloudwatch_utils.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/data_generator.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/db_utils.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/event_monitor.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/exceptions.py create mode 100644 deps/wazuh_testing/wazuh_testing/modules/aws/s3_utils.py create mode 100644 tests/integration/test_aws/README.md create mode 100644 tests/integration/test_aws/conftest.py create mode 100644 tests/integration/test_aws/data/configuration_template/basic_test_module/bucket_configuration_defaults.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/basic_test_module/cloudwatch_configuration_defaults.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/basic_test_module/inspector_configuration_defaults.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/discard_regex_test_module/configuration_discard_regex.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/log_groups_test_module/configuration_log_groups.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_with_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_without_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/cloudwatch_configuration_with_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/inspector_configuration_with_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/service_configuration_without_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_bucket_and_service_missing.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_multiple_bucket_and_service_tags.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_bucket.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_service.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_bucket.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_service.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/path_suffix_test_module/configuration_path_suffix.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/path_test_module/configuration_path.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/regions_test_module/bucket_configuration_regions.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/regions_test_module/cloudwatch_configuration_regions.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/regions_test_module/inspector_configuration_regions.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_from_bucket.yaml create mode 100644 tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_log_stream.yaml create mode 100644 tests/integration/test_aws/data/test_cases/basic_test_module/cases_bucket_defaults.yaml create mode 100644 tests/integration/test_aws/data/test_cases/basic_test_module/cases_cloudwatch_defaults.yaml create mode 100644 tests/integration/test_aws/data/test_cases/basic_test_module/cases_inspector_defaults.yaml create mode 100644 tests/integration/test_aws/data/test_cases/discard_regex_test_module/cases_discard_regex.yaml create mode 100644 tests/integration/test_aws/data/test_cases/log_groups_test_module/cases_log_groups.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_multiple_calls.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_with_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_without_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_multiple_calls.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_with_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_multiple_calls.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_with_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_service_without_only_logs_after.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_bucket_and_service_missing.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_bucket.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_service.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_bucket.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_service.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_multiple_bucket_and_service_tags.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_bucket.yaml create mode 100644 tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_service.yaml create mode 100644 tests/integration/test_aws/data/test_cases/path_suffix_test_module/cases_path_suffix.yaml create mode 100644 tests/integration/test_aws/data/test_cases/path_test_module/cases_path.yaml create mode 100644 tests/integration/test_aws/data/test_cases/regions_test_module/cases_bucket_regions.yaml create mode 100644 tests/integration/test_aws/data/test_cases/regions_test_module/cases_cloudwatch_regions.yaml create mode 100644 tests/integration/test_aws/data/test_cases/regions_test_module/cases_inspector_regions.yaml create mode 100644 tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_from_bucket.yaml create mode 100644 tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_log_streams.yaml create mode 100644 tests/integration/test_aws/test_basic.py create mode 100644 tests/integration/test_aws/test_discard_regex.py create mode 100644 tests/integration/test_aws/test_log_groups.py create mode 100644 tests/integration/test_aws/test_only_logs_after.py create mode 100644 tests/integration/test_aws/test_parser.py create mode 100644 tests/integration/test_aws/test_path.py create mode 100644 tests/integration/test_aws/test_path_suffix.py create mode 100644 tests/integration/test_aws/test_regions.py create mode 100644 tests/integration/test_aws/test_remove_from_bucket.py diff --git a/CHANGELOG.md b/CHANGELOG.md index dfd57373f6..6ed3507da8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ Release report: TBD ### Added +- Add integration tests for AWS module. ([#3911](https://github.com/wazuh/wazuh-qa/pull/3911)) \- (Framework + Tests + Documentation) - Add tests for msu patches with no associated CVE . ([#4009](https://github.com/wazuh/wazuh-qa/pull/4009)) \- (Framework + Tests) - Add tests with new options to avoid FIM synchronization overlapping. ([#3318](https://github.com/wazuh/wazuh-qa/pull/3318)) \- (Framework + tests) - Add Logcollector millisecond granularity support test case ([#3910](https://github.com/wazuh/wazuh-qa/pull/3910)) \- (Tests) diff --git a/deps/wazuh_testing/wazuh_testing/__init__.py b/deps/wazuh_testing/wazuh_testing/__init__.py index 24d541af85..2972b10f57 100644 --- a/deps/wazuh_testing/wazuh_testing/__init__.py +++ b/deps/wazuh_testing/wazuh_testing/__init__.py @@ -40,6 +40,8 @@ WAZUH_TESTING_PATH = os.path.dirname(os.path.abspath(__file__)) WAZUH_TESTING_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') DEFAULT_AUTHD_PASS_PATH = os.path.join(WAZUH_PATH, 'etc', 'authd.pass') +TEMPLATE_DIR = 'configuration_template' +TEST_CASES_DIR = 'test_cases' # Daemons diff --git a/deps/wazuh_testing/wazuh_testing/modules/api/event_monitor.py b/deps/wazuh_testing/wazuh_testing/modules/api/event_monitor.py index 1bbcc8f0f8..de599e7614 100644 --- a/deps/wazuh_testing/wazuh_testing/modules/api/event_monitor.py +++ b/deps/wazuh_testing/wazuh_testing/modules/api/event_monitor.py @@ -35,9 +35,10 @@ def check_api_event(file_monitor=None, callback='', error_message=None, update_p callback (str): log regex to check in the file error_message (str): error message to show in case of expected event does not occur update_position (boolean): filter configuration parameter to search in the file - timeout (str): timeout to check the event in the file + timeout (int): timeout to check the event in the file prefix (str): log pattern regex accum_results (int): Accumulation of matches. + file_to_monitor (str): File to be monitored. """ file_monitor = FileMonitor(file_to_monitor) if file_monitor is None else file_monitor error_message = f"Could not find this event in {file_to_monitor}: {callback}" if error_message is None else \ diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/__init__.py b/deps/wazuh_testing/wazuh_testing/modules/aws/__init__.py new file mode 100644 index 0000000000..04281b5b4e --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/__init__.py @@ -0,0 +1,58 @@ +from pathlib import Path + +from wazuh_testing import WAZUH_PATH + +AWS_MODULE_PATH = Path(WAZUH_PATH, 'wodles', 'aws') +S3_CLOUDTRAIL_DB_PATH = Path(AWS_MODULE_PATH, 's3_cloudtrail.db') +AWS_SERVICES_DB_PATH = Path(AWS_MODULE_PATH, 'aws_services.db') + +AWS_LOGS = 'AWSLogs' +RANDOM_ACCOUNT_ID = '819751203818' +CLOUDTRAIL = 'CloudTrail' +GUARDDUTY = 'GuardDuty' +VPC_FLOW_LOGS = 'vpcflowlogs' +FLOW_LOG_ID = 'fl-0754d951c16f517fa' +CONFIG = 'Config' +ELASTIC_LOAD_BALANCING = 'elasticloadbalancing' +SERVER_ACCESS_TABLE_NAME = 's3_server_access' +PERMANENT_CLOUDWATCH_LOG_GROUP = 'wazuh-cloudwatchlogs-integration-tests' +TEMPORARY_CLOUDWATCH_LOG_GROUP = 'temporary-log-group' +FAKE_CLOUDWATCH_LOG_GROUP = 'fake-log-group' + +EVENT_TIME_FORMAT = '%Y-%m-%dT%H:%M:%SZ' +PATH_DATE_FORMAT = '%Y/%m/%d' +PATH_DATE_NO_PADED_FORMAT = '%Y/%-m/%-d' +FILENAME_DATE_FORMAT = '%Y%m%dT%H%MZ' +ALB_DATE_FORMAT = '%Y-%m-%dT%H:%M:%fZ' + +US_EAST_1_REGION = 'us-east-1' + +JSON_EXT = '.json' +LOG_EXT = '.log' +JSON_GZ_EXT = '.jsonl.gz' +CSV_EXT = '.csv' + +# Bucket types +CLOUD_TRAIL_TYPE = 'cloudtrail' +VPC_FLOW_TYPE = 'vpcflow' +CONFIG_TYPE = 'config' +ALB_TYPE = 'alb' +CLB_TYPE = 'clb' +NLB_TYPE = 'nlb' +KMS_TYPE = 'kms' +MACIE_TYPE = 'macie' +KMS_TYPE = 'kms' +TRUSTED_ADVISOR_TYPE = 'trusted' +CUSTOM_TYPE = 'custom' +GUARD_DUTY_TYPE = 'guardduty' +NATIVE_GUARD_DUTY_TYPE = 'native-guardduty' +WAF_TYPE = 'waf' +SERVER_ACCESS = 'server_access' +CISCO_UMBRELLA_TYPE = 'cisco_umbrella' + +# Params + +ONLY_LOGS_AFTER_PARAM = '--only_logs_after' + + +local_internal_options = {'wazuh_modules.debug': '2', 'monitord.rotate_log': '0'} diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/cli_utils.py b/deps/wazuh_testing/wazuh_testing/modules/aws/cli_utils.py new file mode 100644 index 0000000000..b540a7a73b --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/cli_utils.py @@ -0,0 +1,61 @@ +import subprocess +from pathlib import Path + +from wazuh_testing import logger +from wazuh_testing.modules.aws import AWS_MODULE_PATH +from wazuh_testing.modules.aws.exceptions import OutputAnalysisError + +AWS_BINARY_PATH = Path(AWS_MODULE_PATH, 'aws-s3') + + +def call_aws_module(*parameters): + """Given some parameters call the AWS module and return the output. + + Returns: + str: The command output. + """ + command = [AWS_BINARY_PATH, *parameters] + logger.debug("Calling AWS module with: '%s'", command) + command_result = subprocess.run(command, capture_output=True) + + return command_result.stdout.decode() + + +def _default_callback(line: str): + print(line) + return line + + +def analyze_command_output( + command_output, callback=_default_callback, expected_results=1, error_message='' +): + """Analyze the given command output searching for a pattern. + + Args: + command_output (str): The output to analyze. + callback (Callable, optional): A callback to process each line. Defaults to _default_callback. + expected_results (int, optional): Number of expected results. Defaults to 1. + error_message (str, optional): Message to show with the exception. Defaults to ''. + + Raises: + OutputAnalysisError: When the expected results are not correct. + """ + + results = [] + + for line in command_output.splitlines(): + logger.debug(line) + item = callback(line) + + if item is not None: + results.append(item) + + results_len = len(results) + + if results_len != expected_results: + if error_message: + logger.error(error_message) + logger.error('Results found: %s', results_len) + logger.error('Results expected: %s', expected_results) + raise OutputAnalysisError(error_message) + raise OutputAnalysisError() diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/cloudwatch_utils.py b/deps/wazuh_testing/wazuh_testing/modules/aws/cloudwatch_utils.py new file mode 100644 index 0000000000..a4ea30ed92 --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/cloudwatch_utils.py @@ -0,0 +1,87 @@ +"""AWS CloudWatch related utils""" + +from time import time +from uuid import uuid4 + +import boto3 +from wazuh_testing.modules.aws import PERMANENT_CLOUDWATCH_LOG_GROUP, US_EAST_1_REGION + +session = boto3.Session(profile_name='qa') +logs = session.client('logs', region_name=US_EAST_1_REGION) + + +def create_log_group(log_group_name): + """Create a log group. + + Args: + log_group_name (str): Log group name to create. + """ + logs.create_log_group(logGroupName=log_group_name) + + +def delete_log_group(log_group_name): + """Delete the given log group. + + Args: + log_group_name (str): Log group name to delete. + """ + logs.delete_log_group(logGroupName=log_group_name) + + +def create_log_stream(log_group=PERMANENT_CLOUDWATCH_LOG_GROUP): + """Create a log stream within the given log group. + + Args: + log_group (str, optional): Log group to store the stream. Defaults to PERMANENT_CLOUDWATCH_LOG_GROUP. + + Returns: + str: The name of the created log stream. + """ + log_stream_name = str(uuid4()) + logs.create_log_stream(logGroupName=log_group, logStreamName=log_stream_name) + + return log_stream_name + + +def delete_log_stream(log_stream, log_group=PERMANENT_CLOUDWATCH_LOG_GROUP): + """Delete a log stream from the given log group. + + Args: + log_stream (str): The log stream to delete. + log_group (str, optional): Log group to delete the stream. Defaults to PERMANENT_CLOUDWATCH_LOG_GROUP. + """ + logs.delete_log_stream(logGroupName=log_group, logStreamName=log_stream) + + +def create_log_events(log_stream, log_group=PERMANENT_CLOUDWATCH_LOG_GROUP, event_number=1): + """Create a log event within the given log stream and group. + + Args: + log_stream (str): The log stream to delete. + log_group (str, optional): Log group to delete the stream. Defaults to PERMANENT_CLOUDWATCH_LOG_GROUP. + event_number (int, optional): Number of events to create. Defaults to 1. + """ + + events = [ + {'timestamp': int(time() * 1000), 'message': f"Test event number {i}"} for i in range(event_number) + ] + + logs.put_log_events( + logGroupName=log_group, logStreamName=log_stream, logEvents=events, + ) + + +def log_stream_exists(log_group, log_stream) -> bool: + """Check if a log stream exists in a group. + + Args: + log_group (str): Log group to search within. + log_stream (str): Log stream to search. + + Returns: + bool: True if exists else False + """ + response = logs.describe_log_streams(logGroupName=log_group) + log_streams = [item['logStreamName'] for item in response['logStreams']] + + return log_stream in log_streams diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/data_generator.py b/deps/wazuh_testing/wazuh_testing/modules/aws/data_generator.py new file mode 100644 index 0000000000..aad90f7c2e --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/data_generator.py @@ -0,0 +1,1183 @@ +"""Utils to generate sample data to AWS""" +import csv +import json +from datetime import datetime +from io import StringIO +from os.path import join +from uuid import uuid4 + +from wazuh_testing.modules import aws as cons +from wazuh_testing.tools.utils import get_random_ip, get_random_port, get_random_string + + +def get_random_interface_id(): + """Return a random interface ID that match with the AWS format.""" + return f"eni-{get_random_string(17)}" + + +class DataGenerator: + BASE_PATH = '' + BASE_FILE_NAME = '' + + compress = False + + def get_filename(self, *args, **kwargs): + """Return the filename according to the integration format. + + Returns: + str: Synthetic filename. + """ + raise NotImplementedError() + + def get_data_sample(self, *args, **kwargs): + """Return a sample of data according to the integration format. + + Returns: + dict: Synthetic data. + """ + raise NotImplementedError() + + +class CloudTrailDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.CLOUDTRAIL, cons.US_EAST_1_REGION) + BASE_FILE_NAME = f"{cons.RANDOM_ACCOUNT_ID}_{cons.CLOUDTRAIL}_{cons.US_EAST_1_REGION}_" + + def get_filename(self): + """Return the filename in the cloudtrail format. + + Example: + /AWSLogs////CloudTrail//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}_{abs(hash(now))}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the cloudtrail format. + + Returns: + str: Synthetic data. + """ + return json.dumps({ + 'Records': [ + { + 'eventVersion': '1.08', + 'userIdentity': { + 'type': 'AWSService', + 'invokedBy': 'ec2.amazonaws.com' + }, + 'eventTime': datetime.utcnow().strftime(cons.EVENT_TIME_FORMAT), + 'eventSource': 'sts.amazonaws.com', + 'eventName': 'AssumeRole', + 'awsRegion': cons.US_EAST_1_REGION, + 'sourceIPAddress': 'ec2.amazonaws.com', + 'userAgent': 'ec2.amazonaws.com', + 'requestParameters': { + 'roleArn': f"arn:aws:iam::{cons.RANDOM_ACCOUNT_ID}:role/demo-415-v2-InstanceRole-1FB0FMP2EXOKN", + 'roleSessionName': 'i-0e9ddef5daf05c7df' + }, + 'responseElements': { + 'credentials': { + 'accessKeyId': 'ASIASNL6BLJL7ZA3J6WP', + 'sessionToken': str(uuid4()), + 'expiration': 'Dec 23, 2021, 3:51:35 PM' + } + }, + 'requestID': str(uuid4()), + 'eventID': str(uuid4()), + 'readOnly': True, + 'resources': [ + { + 'accountId': cons.RANDOM_ACCOUNT_ID, + 'type': 'AWS::IAM::Role', + 'ARN': f"arn:aws:iam::{cons.RANDOM_ACCOUNT_ID}:role/demo-415-v2-InstanceRole-1FB0FMP2EXOKN" + } + ], + 'eventType': 'AwsApiCall', + 'managementEvent': True, + 'eventCategory': 'Management', + 'recipientAccountId': cons.RANDOM_ACCOUNT_ID, + 'sharedEventID': str(uuid4()) + } + ] + }) + + +class VPCDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.VPC_FLOW_LOGS, cons.US_EAST_1_REGION) + BASE_FILE_NAME = f"{cons.RANDOM_ACCOUNT_ID}_{cons.VPC_FLOW_LOGS}_{cons.US_EAST_1_REGION}_" + + def get_filename(self): + """Return the filename in the VPC format. + + Example: + /AWSLogs////vpcflowlogs//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = ( + f"{self.BASE_FILE_NAME}{cons.FLOW_LOG_ID}_{now.strftime(cons.FILENAME_DATE_FORMAT)}_{abs(hash(now))}" + f"{cons.LOG_EXT}" + ) + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the VPC format. + + Returns: + str: Synthetic data. + """ + data = [ + [ + 'version', 'account-id', 'interface-id', 'srcaddr', 'dstaddr', 'srcport', 'dstport', 'protocol', + 'packets', 'bytes', 'start', 'end', 'action', 'log-status' + ] + ] + + for _ in range(5): + data.append( + [ + '2', cons.RANDOM_ACCOUNT_ID, get_random_interface_id(), get_random_ip(), get_random_ip(), + get_random_port(), get_random_port(), '6', '39', '4698', '1622505433', '1622505730', 'ACCEPT', 'OK' + ] + ) + buffer = StringIO() + csv.writer(buffer, delimiter=' ').writerows(data) + + return buffer.getvalue() + + +class ConfigDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.CONFIG, cons.US_EAST_1_REGION) + BASE_FILE_NAME = f"{cons.RANDOM_ACCOUNT_ID}_{cons.CONFIG}_{cons.US_EAST_1_REGION}_ConfigHistory_AWS_" + + def get_filename(self): + """Return the filename in the Config format. + + Example: + /AWSLogs////Config//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_NO_PADED_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}_{abs(hash(now))}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the Config format. + + Returns: + str: Synthetic data. + """ + return json.dumps({ + 'fileVersion': '1.0', + 'configurationItems': [ + { + 'relatedEvents': [], + 'relationships': [ + { + 'resourceId': f"vol-{get_random_string(17)}", + 'resourceType': 'AWS::EC2::Volume', + 'name': 'Is associated with ' + } + ], + 'configuration': { + 'complianceType': 'NON_COMPLIANT', + 'targetResourceId': f"vol-{get_random_string(17)}", + 'targetResourceType': 'AWS::EC2::Volume', + 'configRuleList': [ + { + 'configRuleArn': ( + f"arn:aws:config:us-east-1:{cons.RANDOM_ACCOUNT_ID}:config-rule/" + 'config-rule-b1eqqf'), + 'configRuleId': 'config-rule-b1eqqf', + 'configRuleName': 'encrypted-volumes', + 'complianceType': 'NON_COMPLIANT' + } + ] + }, + 'supplementaryConfiguration': {}, + 'tags': {}, + 'configurationItemVersion': '1.3', + 'configurationItemCaptureTime': '2020-06-01T02:12:37.713Z', + 'configurationStateId': 1590977557713, + 'awsAccountId': cons.RANDOM_ACCOUNT_ID, + 'configurationItemStatus': 'ResourceDiscovered', + 'resourceType': 'AWS::Config::ResourceCompliance', + 'resourceId': f"AWS::EC2::Volume/vol-{get_random_string(17)}", + 'awsRegion': 'us-east-1', + 'configurationStateMd5Hash': '' + } + ] + }) + + +class ALBDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.ELASTIC_LOAD_BALANCING, cons.US_EAST_1_REGION) + BASE_FILE_NAME = f"{cons.RANDOM_ACCOUNT_ID}_{cons.ELASTIC_LOAD_BALANCING}_{cons.US_EAST_1_REGION}_" + + def get_filename(self): + """Return the filename in the ALB format. + + Example: + /AWSLogs////elasticloadbalancing//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = ( + f"{self.BASE_FILE_NAME}_app.ALB-qatests_{now.strftime(cons.FILENAME_DATE_FORMAT)}_{abs(hash(now))}_" + f"{get_random_ip()}_pczeay_{cons.LOG_EXT}" + ) + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the ALB format. + + Returns: + str: Synthetic data. + """ + now = datetime.utcnow() + data = [] + + for _ in range(5): + data.append( + [ + 'http', # type + now.strftime(cons.ALB_DATE_FORMAT), # time + 'app/ALB-qatests', # elb + f"{get_random_ip()}:{get_random_port()}", # client:port + f"{get_random_ip()}:{get_random_port()}", # target:port + 0.001, # request_processing_time + 0.001, # target_processing_time + 0.000, # response_processing_time + 403, # elb_status_code + 403, # target_status_code + 136, # received_bytes + 5173, # sent_bytes + f"GET http://{get_random_ip()}:80/ HTTP/1.1", # request + 'Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)', # user_agent + '-', # ssl_cipher + '-', # ssl_protocol + # target_group_arn + f"arn:aws:elasticloadbalancing:{cons.US_EAST_1_REGION}:{cons.RANDOM_ACCOUNT_ID}:targetgroup/EC2/", + f"Root=1-5fbc4c52-{get_random_string(24)}", # trace_id + '-', # domain_name + '-', # chosen_cert_arn + 0, # matched_rule_priority + now.strftime(cons.ALB_DATE_FORMAT), # request_creation_time + 'forward', # actions_executed + '-', # redirect_url + '-', # error_reason + f"{get_random_ip()}:{get_random_port()} {get_random_ip()}:{get_random_port()}", # target:port_list + '403', # target_status_code_list + '-', # classification + '-' # classification_reason + ] + ) + buffer = StringIO() + csv.writer(buffer, delimiter=' ').writerows(data) + + return buffer.getvalue() + + +class CLBDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.ELASTIC_LOAD_BALANCING, cons.US_EAST_1_REGION) + BASE_FILE_NAME = f"{cons.RANDOM_ACCOUNT_ID}_{cons.ELASTIC_LOAD_BALANCING}_{cons.US_EAST_1_REGION}_" + + def get_filename(self): + """Return the filename in the CLB format. + + Example: + /AWSLogs////elasticloadbalancing//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = ( + f"{self.BASE_FILE_NAME}qatests-APIClassi_{now.strftime(cons.FILENAME_DATE_FORMAT)}_{abs(hash(now))}_" + f"{get_random_ip()}{cons.LOG_EXT}" + ) + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the CLB format. + + Returns: + str: Synthetic data. + """ + now = datetime.utcnow() + data = [] + for _ in range(5): + data.append( + [ + now.strftime(cons.ALB_DATE_FORMAT), # time + 'qatests-APIClassi', # elb + f"{get_random_ip()}:{get_random_port()}", # client:port + f"{get_random_ip()}:{get_random_port()}", # backend:port + 0.000628, # request_processing_time + 0.001, # backend_processing_time + 0.000015, # response_processing_time + 403, # elb_status_code + 403, # backend_status_code + 1071, # received_bytes + 2250, # sent_bytes + '- - -', # request + 'Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)', # user_agent + '-', # ssl_cipher + '-', # ssl_protocol + ] + ) + buffer = StringIO() + csv.writer(buffer, delimiter=' ').writerows(data) + + return buffer.getvalue() + + +class NLBDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.ELASTIC_LOAD_BALANCING, cons.US_EAST_1_REGION) + BASE_FILE_NAME = f"{cons.RANDOM_ACCOUNT_ID}_{cons.ELASTIC_LOAD_BALANCING}_{cons.US_EAST_1_REGION}_" + + def get_filename(self): + """Return the filename in the NLB format. + + Example: + /AWSLogs////elasticloadbalancing//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = ( + f"{self.BASE_FILE_NAME}net.qatests_{now.strftime(cons.FILENAME_DATE_FORMAT)}_{abs(hash(now))}_" + f"{get_random_ip()}{cons.LOG_EXT}" + ) + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the NLB format. + + Returns: + str: Synthetic data. + """ + now = datetime.utcnow() + data = [] + for _ in range(5): + data.append( + [ + 'tls', # type + '2.0', # version + now.strftime(cons.ALB_DATE_FORMAT), # time + 'net/qatests', # elb + get_random_string(16), # listener + f"{get_random_ip()}:{get_random_port()}", # client:port + f"{get_random_ip()}:{get_random_port()}", # destination:port + 17553, # connection_time + 0.001, # tls_handshake_time + 1071, # received_bytes + 2250, # sent_bytes + '-', # incoming_tls_alert + '-', # chosen_cert_arn + '-', # chosen_cert_serial + '-', # tls_cipher + '-', # tls_protocol_version + '-', # tls_named_group + '-', # domain_name + '-', # alpn_fe_protocol + '-', # alpn_be_protocol + '-', # alpn_client_preference_list + ] + ) + buffer = StringIO() + csv.writer(buffer, delimiter=' ').writerows(data) + + return buffer.getvalue() + + +class KMSDataGenerator(DataGenerator): + BASE_PATH = '' + BASE_FILE_NAME = 'firehose_kms-1-' + + def get_filename(self): + """Return the filename in the KMS format. + + Example: + /// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}_{str(uuid4())}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the KMS format. + + Returns: + str: Synthetic data. + """ + return json.dumps( + { + 'version': '0', + 'id': str(uuid4()), + 'detail-type': 'AWS API Call via CloudTrail', + 'source': 'aws.kms', + 'account': cons.RANDOM_ACCOUNT_ID, + 'time': '2018-11-07T17:27:01Z', + 'region': cons.US_EAST_1_REGION, + 'resources': [], + 'detail': { + 'eventVersion': '1.05', + 'userIdentity': { + 'type': 'IAMUser', + 'principalId': get_random_string(20), + 'arn': f"arn:aws:iam::{cons.RANDOM_ACCOUNT_ID}:user/fake.user", + 'accountId': cons.RANDOM_ACCOUNT_ID, + 'accessKeyId': get_random_string(20), + 'userName': 'fake.user', + 'sessionContext': { + 'attributes': { + 'mfaAuthenticated': 'false', + 'creationDate': '2018-11-07T07:53:47Z' + } + }, + 'invokedBy': 'secretsmanager.amazonaws.com' + }, + 'eventTime': '2018-11-07T17:27:01Z', + 'eventSource': 'kms.amazonaws.com', + 'eventName': 'GenerateDataKey', + 'awsRegion': cons.RANDOM_ACCOUNT_ID, + 'sourceIPAddress': 'secretsmanager.amazonaws.com', + 'userAgent': 'secretsmanager.amazonaws.com', + 'requestParameters': { + 'keySpec': 'AES_256', + 'encryptionContext': { + 'SecretARN': f"arn:aws:secretsmanager:us-east-1:{cons.RANDOM_ACCOUNT_ID}:secret:test-aws", + 'SecretVersionId': str(uuid4()) + }, + 'keyId': 'alias/aws/secretsmanager' + }, + 'responseElements': None, + 'requestID': str(uuid4()), + 'eventID': str(uuid4()), + 'readOnly': True, + 'resources': [ + { + 'ARN': f"arn:aws:kms:us-east-1:{cons.RANDOM_ACCOUNT_ID}:key/{str(uuid4())}", + 'accountId': cons.RANDOM_ACCOUNT_ID, + 'type': 'AWS::KMS::Key' + } + ], + 'eventType': 'AwsApiCall', + 'vpcEndpointId': f"vpce-{get_random_string(17)}" + } + } + ) + + +class MacieDataGenerator(DataGenerator): + BASE_PATH = '' + BASE_FILE_NAME = 'firehose_macie-1-' + + def get_filename(self): + """Return the filename in the Macie format. + + Example: + /// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}_{str(uuid4())}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the Macie format. + + Returns: + str: Synthetic data. + """ + + return json.dumps( + { + 'version': '0', + 'id': str(uuid4()), + 'detail-type': 'Macie Alert', + 'source': 'aws.macie', + 'account': cons.RANDOM_ACCOUNT_ID, + 'time': '2021-01-01T00:20:42Z', + 'region': 'us-east-1', + 'resources': [ + f"arn:aws:macie:us-east-1:{cons.RANDOM_ACCOUNT_ID}:trigger/{str(uuid4())}/alert", + f"arn:aws:macie:us-east-1:{cons.RANDOM_ACCOUNT_ID}:trigger/{str(uuid4())}" + ], + 'detail': { + 'notification-type': 'ALERT_CREATED', + 'tags': [ + 'Open Permissions', + 'Basic Alert' + ], + 'name': 'S3 Bucket IAM policy grants global read rights', + 'severity': 'CRITICAL', + 'url': 'https://mt.us-east-1.macie.aws.amazon.com/posts/arn%3Aaws%3Amacie%3Aus-east-1', + 'alert-arn': f"arn:aws:macie:us-east-1:{cons.RANDOM_ACCOUNT_ID}:trigger/{str(uuid4())}/alert", + 'risk-score': 9, + 'created-at': '2021-01-01T00:20:42.364509', + 'actor': 'resources.wazuh.com', + 'summary': { + 'Description': 'S3 Bucket uses IAM policy to grant read rights to Everyone.', + 'Bucket': { + 'resources.wazuh.com': 1 + }, + 'Record Count': 1, + 'ACL': { + 'resources.wazuh.com': [ + { + 'Owner': { + 'DisplayName': 'wazuh', + 'ID': get_random_string(64), + }, + 'Grants': [ + { + 'Grantee': { + 'Type': 'CanonicalUser', + 'DisplayName': 'wazuh', + 'ID': get_random_string(64), + }, + 'Permission': 'FULL_CONTROL' + }, + { + 'Grantee': { + 'Type': 'Group', + 'URI': 'http://acs.amazonaws.com/groups/global/AllUsers' + }, + 'Permission': 'READ' + } + ] + } + ] + }, + 'Event Count': 1, + 'Timestamps': { + '2021-01-01T00:11:49.171020Z': 1 + }, + 'recipientAccountId': { + cons.RANDOM_ACCOUNT_ID: 1 + } + }, + 'trigger': { + 'rule-arn': ( + f"arn:aws:macie:us-east-1:{cons.RANDOM_ACCOUNT_ID}:trigger/b731d9ffb1fe61508d4a478c92efa666" + ), + 'alert-type': 'basic', + 'created-at': '2020-12-29 16:36:17.412000+00:00', + 'description': 'S3 Bucket uses IAM policy to grant read rights to Everyone.', + 'risk': 9 + } + } + } + ) + + +class TrustedAdvisorDataGenerator(DataGenerator): + BASE_PATH = '' + BASE_FILE_NAME = 'firehose_trustedadvisor-1-' + + def get_filename(self): + """Return the filename in the Trusted Advisor format. + + Example: + /// + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the Trusted Advisor format. + + Returns: + str: Synthetic data. + """ + return json.dumps( + { + 'version': '0', + 'id': get_random_string(26), + 'detail-type': 'Trusted Advisor Check Item Refresh Notification', + 'source': 'aws.trustedadvisor', + 'account': cons.RANDOM_ACCOUNT_ID, + 'time': datetime.utcnow().strftime(cons.FILENAME_DATE_FORMAT), + 'region': 'us-east-1', + 'resources': [], + 'detail': { + 'check-name': 'IAM Group', + 'check-item-detail': { + 'Status': 'Green', + 'Current Usage': '1', + 'Limit Name': 'Groups', + 'Region': '-', + 'Service': 'IAM', + 'Limit Amount': '300' + }, + 'status': 'OK', + 'resource_id': '', + 'uuid': str(uuid4()) + } + } + ) + + +class GuardDutyDataGenerator(DataGenerator): + BASE_PATH = '' + BASE_FILE_NAME = 'firehose_guardduty-1-' + + def get_filename(self): + """Return the filename in the Guard Duty format. + + Example: + /// + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the Guard Duty format. + + Returns: + str: Synthetic data. + """ + return json.dumps( + { + 'version': '0', + 'id': str(uuid4()), + 'detail-type': 'GuardDuty Finding', + 'source': 'aws.guardduty', + 'account': cons.RANDOM_ACCOUNT_ID, + 'time': '2021-07-08T03:45:04Z', + 'region': 'us-east-1', + 'resources': [], + 'detail': { + 'schemaVersion': '2.0', + 'accountId': cons.RANDOM_ACCOUNT_ID, + 'region': 'us-east-1', + 'partition': 'aws', + 'id': 'e8bc77e2d65ffa20de95cc6e7a94e926', + 'arn': f"arn:aws:guardduty:us-east-1:{cons.RANDOM_ACCOUNT_ID}:detector/", + 'type': 'Recon:EC2/PortProbeUnprotectedPort', + 'resource': { + 'resourceType': 'Instance', + 'instanceDetails': { + 'instanceId': f"i-{get_random_string(8)}", + 'instanceType': 't2.micro', + 'launchTime': '2014-12-30T18:46:13Z', + 'platform': None, + 'productCodes': [], + 'iamInstanceProfile': None, + 'networkInterfaces': [ + { + 'ipv6Addresses': [], + 'networkInterfaceId': f"eni-{get_random_string(8)}", + 'privateDnsName': 'ip-10-0-0-250.ec2.internal', + 'privateIpAddress': get_random_ip(), + 'privateIpAddresses': [ + { + 'privateDnsName': 'ip-10-0-0-250.ec2.internal', + 'privateIpAddress': get_random_ip() + } + ], + 'subnetId': 'subnet-6b1d6203', + 'vpcId': f"vpc-{get_random_string(8)}", + 'securityGroups': [ + { + 'groupName': 'default', + 'groupId': f"sg-{get_random_string(8)}" + } + ], + 'publicDnsName': 'ec2-105-71-92-143.compute-1.amazonaws.com', + 'publicIp': get_random_ip() + } + ], + 'outpostArn': None, + 'tags': [ + { + 'key': 'service_name', + 'value': 'vpn' + }, + { + 'key': 'Name', + 'value': 'vpn-gateway (r)' + } + ], + 'instanceState': 'running', + 'availabilityZone': 'us-east-1e', + 'imageId': f"ami-{get_random_string(8)}", + 'imageDescription': 'None' + } + }, + 'service': { + 'serviceName': 'guardduty', + 'detectorId': str(uuid4()), + 'action': { + 'actionType': 'PORT_PROBE', + 'portProbeAction': { + 'portProbeDetails': [ + { + 'localPortDetails': { + 'port': 1723, + 'portName': 'Unknown' + }, + 'remoteIpDetails': { + 'ipAddressV4': get_random_ip(), + 'organization': { + 'asn': '211680', + 'asnOrg': 'Sistemas Informaticos, S.A.', + 'isp': 'Sistemas Informaticos, S.A.', + 'org': 'Sistemas Informaticos, S.A.' + }, + 'country': { + 'countryName': 'Portugal' + }, + 'city': { + 'cityName': '' + }, + 'geoLocation': { + 'lat': 38.7057, + 'lon': -9.1359 + } + } + } + ], + 'blocked': False + } + }, + 'resourceRole': 'TARGET', + 'additionalInfo': { + 'threatName': 'Scanner', + 'threatListName': 'ProofPoint' + }, + 'evidence': { + 'threatIntelligenceDetails': [ + { + 'threatNames': [ + 'Scanner' + ], + 'threatListName': 'ProofPoint' + } + ] + }, + 'eventFirstSeen': '2021-04-20T14:40:04Z', + 'eventLastSeen': '2021-07-08T03:15:41Z', + 'archived': False, + 'count': 5 + }, + 'severity': 2, + 'createdAt': '2021-04-20T14:53:32.735Z', + 'updatedAt': '2021-07-08T03:31:04.017Z', + 'title': 'Unprotected port on EC2 instance i-3bf6a5c5 is being probed.', + 'description': ( + 'EC2 instance has an unprotected port which is being probed by a known malicious host.' + ) + } + } + ) + + +class NativeGuardDutyDataGenerator(DataGenerator): + BASE_PATH = join(cons.AWS_LOGS, cons.RANDOM_ACCOUNT_ID, cons.GUARDDUTY, cons.US_EAST_1_REGION) + BASE_FILE_NAME = '' + + compress = True + + def get_filename(self): + """Return the filename in the Native Guard Duty format. + + Example: + /AWSLogs///GuardDuty//// + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{str(uuid4())}{cons.JSON_GZ_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the Native Guard Duty format. + + Returns: + str: Synthetic data. + """ + random_ip = get_random_ip() + return json.dumps( + { + 'schemaVersion': '2.0', + 'accountId': cons.RANDOM_ACCOUNT_ID, + 'region': 'us-east-1', + 'partition': 'aws', + 'id': '3ac1fd234445e957d526a10c72631c8f', + 'arn': f"arn:aws:guardduty:us-east-1:{cons.RANDOM_ACCOUNT_ID}:detector/c0bfff53bb19fbee16ed05a0b21d3b/", + 'type': 'UnauthorizedAccess:EC2/SSHBruteForce', + 'resource': { + 'resourceType': 'Instance', + 'instanceDetails': { + 'instanceId': f"i-{get_random_string(18)}", + 'instanceType': 'c5.large', + 'launchTime': '2022-10-19T16:17:42.000Z', + 'platform': None, + 'productCodes': [], + 'iamInstanceProfile': None, + 'networkInterfaces': [ + { + 'ipv6Addresses': [], + 'networkInterfaceId': f"eni-{get_random_string(18)}", + 'privateDnsName': f"ip-{random_ip.replace('.', '-')}.ec2.internal", + 'privateIpAddress': random_ip, + 'privateIpAddresses': [ + { + 'privateDnsName': f"ip-{random_ip.replace('.', '-')}.ec2.internal", + 'privateIpAddress': random_ip + } + ], + 'subnetId': f"subnet-{get_random_string(8)}", + 'vpcId': 'vpc-f825c385', + 'securityGroups': [ + { + 'groupName': 'test-ansible', + 'groupId': f"sg-{get_random_string(16)}" + } + ], + 'publicDnsName': f"ec2-{random_ip.replace('.', '-')}.compute-1.amazonaws.com", + 'publicIp': random_ip + } + ], + 'outpostArn': None, + 'tags': [ + { + 'key': 'Name', + 'value': 'some-test-server-investigating' + } + ], + 'instanceState': 'running', + 'availabilityZone': 'us-east-1d', + 'imageId': 'ami-026b57f3c383c2eec', + 'imageDescription': 'Amazon Linux 2 Kernel 5.10 AMI 2.0.20220912.1 x86_64 HVM gp2' + } + }, + 'service': { + 'serviceName': 'guardduty', + 'detectorId': 'c0bfff53bb19fbee16ed05a0b21d3be3', + 'action': { + 'actionType': 'NETWORK_CONNECTION', + 'networkConnectionAction': { + 'connectionDirection': 'INBOUND', + 'remoteIpDetails': { + 'ipAddressV4': random_ip, + 'organization': { + 'asn': '3462', + 'asnOrg': 'Data Communication Business Group', + 'isp': 'Chunghwa Telecom', + 'org': 'Chunghwa Telecom' + }, + 'country': { + 'countryName': 'Taiwan' + }, + 'city': { + 'cityName': 'Tainan City' + }, + 'geoLocation': { + 'lat': 22.9917, + 'lon': 120.2148 + } + }, + 'remotePortDetails': { + 'port': get_random_port(), + 'portName': 'Unknown' + }, + 'localPortDetails': { + 'port': 22, + 'portName': 'SSH' + }, + 'protocol': 'TCP', + 'blocked': False, + 'localIpDetails': { + 'ipAddressV4': random_ip + } + } + }, + 'resourceRole': 'TARGET', + 'additionalInfo': { + 'value': '{}', + 'type': 'default' + }, + 'eventFirstSeen': '2022-10-21T11:14:59.000Z', + 'eventLastSeen': '2022-10-21T11:19:24.000Z', + 'archived': False, + 'count': 1 + }, + 'severity': 2, + 'createdAt': '2022-10-21T11:21:10.027Z', + 'updatedAt': '2022-10-21T11:21:10.027Z', + 'title': f"{get_random_ip()} is performing SSH brute force attacks against i-08cb1e1f2bcce.", + 'description': f"{get_random_ip()} is performing SSH brute force attacks against i-08cb1ef2bcce.f" + } + ) + '\n' + + +class WAFDataGenerator(DataGenerator): + BASE_PATH = '' + BASE_FILE_NAME = 'aws-waf-logs-delivery-stream-1-' + + def get_filename(self): + """Return the filename in the KMS format. + + Example: + /// + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime(cons.PATH_DATE_FORMAT)) + name = f"{self.BASE_FILE_NAME}{now.strftime(cons.FILENAME_DATE_FORMAT)}{cons.JSON_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the cloudtrail format. + + Returns: + str: Synthetic data. + """ + return json.dumps( + { + 'timestamp': 1576280412771, + 'formatVersion': 1, + 'webaclId': ( + f"arn:aws:wafv2:ap-southeast-2:{cons.RANDOM_ACCOUNT_ID}:regional/" + 'webacl/STMTest/1EXAMPLE-2ARN-3ARN-4ARN-123456EXAMPLE' + ), + 'terminatingRuleId': 'STMTest_SQLi_XSS', + 'terminatingRuleType': 'REGULAR', + 'action': 'BLOCK', + 'terminatingRuleMatchDetails': [ + { + 'conditionType': 'SQL_INJECTION', + 'sensitivityLevel': 'HIGH', + 'location': 'HEADER', + 'matchedData': [ + '10', + 'AND', + '1' + ] + } + ], + 'httpSourceName': '-', + 'httpSourceId': '-', + 'ruleGroupList': [], + 'rateBasedRuleList': [], + 'nonTerminatingMatchingRules': [], + 'httpRequest': { + 'clientIp': get_random_ip(), + 'country': 'AU', + 'headers': [ + { + 'name': 'Host', + 'value': 'localhost:1989' + }, + { + 'name': 'User-Agent', + 'value': 'curl/7.61.1' + }, + { + 'name': 'Accept', + 'value': '*/*' + }, + { + 'name': 'x-stm-test', + 'value': '10 AND 1=1' + } + ], + 'uri': '/myUri', + 'args': '', + 'httpVersion': 'HTTP/1.1', + 'httpMethod': 'GET', + 'requestId': 'rid' + }, + 'labels': [ + { + 'name': 'value' + } + ] + } + ) + + +class ServerAccessDataGenerator(DataGenerator): + BASE_PATH = '' + BASE_FILE_NAME = '' + + def get_filename(self): + """Return the filename in the server access format. + + Example: + / + + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + date_format = '%Y-%m-%d-%H-%M-%S' + name = f"{now.strftime(date_format)}-{get_random_string(16).upper()}" + return join(self.BASE_PATH, name) + + def get_data_sample(self): + """Return a sample of data according to the server access format. + + Returns: + str: Synthetic data. + """ + data = [] + + for _ in range(5): + data.append( + [ + str(uuid4()), 'wazuh-server-access-integration-tests', + datetime.utcnow().strftime('[%d/%b/%Y:%H:%M:%S %z]'), get_random_ip(), + f"arn:aws:iam::{cons.RANDOM_ACCOUNT_ID}:user/fake.user", get_random_string(16).upper(), + 'REST.GET.WEBSITE', '-', 'GET, /wazuh-server-access-integration-tests?website= HTTP/1.1', + '404', 'NoSuchWebsiteConfiguration', '343', '-', '85', '-', '-', + ( + 'S3Console/0.4, aws-internal/3 aws-sdk-java/1.11.991' + 'Linux/4.9.230-0.1.ac.224.84.332.metal1.x86_64' + 'OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282 vendor/Oracle_Corporation' + 'cfg/retry-mode/legacy' + ), + '-', str(uuid4()), 'SigV4', 'ECDHE-RSA-AES128-GCM-SHA256', 'AuthHeader', 's3.amazonaws.com', + 'TLSv1.2' + + ] + ) + buffer = StringIO() + csv.writer(buffer, delimiter=' ').writerows(data) + + return buffer.getvalue() + + +class UmbrellaDataGenerator(DataGenerator): + BASE_PATH = 'dnslogs' + BASE_FILE_NAME = '' + + def get_filename(self): + """Return the filename in the umbrella format. + + Example: + /-- + Returns: + str: Synthetic filename. + """ + now = datetime.utcnow() + path = join(self.BASE_PATH, now.strftime('%Y-%m-%d')) + name = f"{self.BASE_FILE_NAME}{now.strftime('%Y-%m-%d')}-00-00-ioxa{cons.CSV_EXT}" + + return join(path, name) + + def get_data_sample(self): + """Return a sample of data according to the cloudtrail format. + + Returns: + str: Synthetic data. + """ + data = [] + + for _ in range(5): + data.append( + [ + datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S'), + 'ActiveDirectoryUserName', + 'ActiveDirectoryUserName,ADSite,Network', + get_random_ip(), + get_random_ip(), + 'Allowed', + '1 (A)', + 'NOERROR', + 'domain-visited.com.', + 'Chat,Photo Sharing,Social Networking,Allow List' + ] + ) + buffer = StringIO() + csv.writer(buffer).writerows(data) + + return buffer.getvalue() + + +# Maps bucket type with corresponding data generator +buckets_data_mapping = { + cons.CLOUD_TRAIL_TYPE: CloudTrailDataGenerator, + cons.VPC_FLOW_TYPE: VPCDataGenerator, + cons.CONFIG_TYPE: ConfigDataGenerator, + cons.ALB_TYPE: ALBDataGenerator, + cons.CLB_TYPE: CLBDataGenerator, + cons.NLB_TYPE: NLBDataGenerator, + cons.KMS_TYPE: KMSDataGenerator, + cons.MACIE_TYPE: MacieDataGenerator, + cons.TRUSTED_ADVISOR_TYPE: TrustedAdvisorDataGenerator, + cons.GUARD_DUTY_TYPE: GuardDutyDataGenerator, + cons.NATIVE_GUARD_DUTY_TYPE: NativeGuardDutyDataGenerator, + cons.WAF_TYPE: WAFDataGenerator, + cons.SERVER_ACCESS: ServerAccessDataGenerator, + cons.CISCO_UMBRELLA_TYPE: UmbrellaDataGenerator +} + + +def get_data_generator(bucket_type, bucket_name): + """Given the bucket type return the correspondant data generator instance. + + Args: + bucket_type (str): Bucket type to match the data generator. + bucket_name (str): Bucket name to match in case of custom or guardduty types. + + Returns: + DataGenerator: Data generator for the given bucket. + """ + if bucket_type == cons.CUSTOM_TYPE: + bucket_type = bucket_name.split('-')[1] + elif bucket_type == cons.GUARD_DUTY_TYPE and 'native' in bucket_name: + bucket_type = cons.NATIVE_GUARD_DUTY_TYPE + + return buckets_data_mapping[bucket_type]() diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/db_utils.py b/deps/wazuh_testing/wazuh_testing/modules/aws/db_utils.py new file mode 100644 index 0000000000..274340ca46 --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/db_utils.py @@ -0,0 +1,256 @@ +import sqlite3 +from collections import namedtuple + +from wazuh_testing.modules.aws import ( + ALB_TYPE, + AWS_SERVICES_DB_PATH, + CISCO_UMBRELLA_TYPE, + CLB_TYPE, + CLOUD_TRAIL_TYPE, + CUSTOM_TYPE, + GUARD_DUTY_TYPE, + NLB_TYPE, + S3_CLOUDTRAIL_DB_PATH, + SERVER_ACCESS_TABLE_NAME, + VPC_FLOW_TYPE, + WAF_TYPE, +) + +SELECT_QUERY_TEMPLATE = 'SELECT * FROM {table_name}' + +S3CloudTrailRow = namedtuple( + 'S3CloudTrailRow', 'bucket_path aws_account_id aws_region log_key processed_date created_date' +) + +S3VPCFlowRow = namedtuple( + 'S3VPCFlowRow', 'bucket_path aws_account_id aws_region flowlog_id log_key processed_date created_date' +) + +S3ALBRow = namedtuple( + 'S3ALBRow', 'bucket_path aws_account_id log_key processed_date created_date' +) + +S3CustomRow = namedtuple( + 'S3CustomRow', 'bucket_path aws_account_id log_key processed_date created_date' +) + +S3GuardDutyRow = namedtuple( + 'S3GuardDutyRow', 'bucket_path aws_account_id log_key processed_date created_date' +) + +S3WAFRow = namedtuple( + 'S3WAFRow', 'bucket_path aws_account_id log_key processed_date created_date' +) + +S3ServerAccessRow = namedtuple( + 'S3ServerAccessRow', 'bucket_path aws_account_id log_key processed_date created_date' +) + +ServiceInspectorRow = namedtuple( + 'ServiceInspectorRow', 'service account_id region timestamp' +) + +ServiceCloudWatchRow = namedtuple( + 'ServiceCloudWatchRow', 'aws_region aws_log_group aws_log_stream next_token start_time end_time' +) + +S3UmbrellaRow = namedtuple( + 'S3UmbrellaRow', 'bucket_path aws_account_id log_key processed_date created_date' +) + +s3_rows_map = { + CLOUD_TRAIL_TYPE: S3CloudTrailRow, + VPC_FLOW_TYPE: S3VPCFlowRow, + ALB_TYPE: S3ALBRow, + CLB_TYPE: S3ALBRow, + NLB_TYPE: S3ALBRow, + CUSTOM_TYPE: S3CustomRow, + GUARD_DUTY_TYPE: S3GuardDutyRow, + WAF_TYPE: S3WAFRow, + SERVER_ACCESS_TABLE_NAME: S3ServerAccessRow, + CISCO_UMBRELLA_TYPE: S3UmbrellaRow +} + +service_rows_map = { + 'cloudwatch_logs': ServiceCloudWatchRow, + 'aws_services': ServiceInspectorRow +} + + +def _get_s3_row_type(bucket_type): + """Get row type for bucket integration. + + Args: + bucket_type (str): The name of the bucket. + + Returns: + Type[S3CloudTrailRow]: The type that match or a default one. + """ + return s3_rows_map.get(bucket_type, S3CloudTrailRow) + + +def _get_service_row_type(table_name): + """Get row type for service integration. + + Args: + table_name (str): Table name to match. + + Returns: + Type[ServiceCloudWatchRow]: The type that match or a default one. + """ + return service_rows_map.get(table_name, ServiceCloudWatchRow) + + +def get_db_connection(path): + """Get an open DB connection. + + Args: + path (Path): The path of the sqlite file. + + Returns: + sqlite3.Connection: A connection with the specified DB. + """ + return sqlite3.connect(path) + + +def table_exists(table_name, db_path=S3_CLOUDTRAIL_DB_PATH): + """Check if the given table name exists. + + Args: + table_name (str): Table name to search for. + + Returns: + bool: True if exists else False. + """ + connection = get_db_connection(db_path) + cursor = connection.cursor() + query = """ + SELECT + name + FROM + sqlite_master + WHERE + type ='table' AND + name NOT LIKE 'sqlite_%'; + """ + + return table_name in [result[0] for result in cursor.execute(query).fetchall()] + + +# cloudtrail.db utils + + +def s3_db_exists(): + """Check if `s3_cloudtrail.db` exists. + + Returns: + bool: True if exists else False. + """ + return S3_CLOUDTRAIL_DB_PATH.exists() + + +def delete_s3_db() -> None: + """Delete `s3_cloudtrail.db` file.""" + if s3_db_exists(): + S3_CLOUDTRAIL_DB_PATH.unlink() + + +def get_s3_db_row(table_name) -> S3CloudTrailRow: + """Return one row from the given table name. + + Args: + table_name (str): Table name to search into. + + Returns: + S3CloudTrailRow: The first row of the table. + """ + connection = get_db_connection(S3_CLOUDTRAIL_DB_PATH) + cursor = connection.cursor() + result = cursor.execute(SELECT_QUERY_TEMPLATE.format(table_name=table_name)).fetchone() + row_type = _get_s3_row_type(table_name) + return row_type(*result) + + +def get_multiple_s3_db_row(table_name): + """Return all rows from the given table name. + + Args: + table_name (str): Table name to search into. + + Yields: + Iterator[S3CloudTrailRow]: All the rows in the table. + """ + connection = get_db_connection(S3_CLOUDTRAIL_DB_PATH) + cursor = connection.cursor() + row_type = _get_s3_row_type(table_name) + + for row in cursor.execute(SELECT_QUERY_TEMPLATE.format(table_name=table_name)): + yield row_type(*row) + + +def table_exists_or_has_values(table_name, db_path=S3_CLOUDTRAIL_DB_PATH): + """Check if the given table name exists. If exists check if has values. + + Args: + table_name (str): Table name to search for. + + Returns: + bool: True if exists or has values else False. + """ + connection = get_db_connection(db_path) + cursor = connection.cursor() + try: + return bool(cursor.execute(SELECT_QUERY_TEMPLATE.format(table_name=table_name)).fetchall()) + except sqlite3.OperationalError: + return False + + +# aws_services.db utils + +def services_db_exists(): + """Check if `aws_services.db` exists. + + Returns: + bool: True if exists else False. + """ + return AWS_SERVICES_DB_PATH.exists() + + +def delete_services_db() -> None: + """Delete `aws_services.db` file.""" + if services_db_exists(): + AWS_SERVICES_DB_PATH.unlink() + + +def get_service_db_row(table_name): + """Return one row from the given table name. + + Args: + table_name (str): Table name to search into. + + Returns: + ServiceInspectorRow: The first row of the table. + """ + row_type = _get_service_row_type(table_name) + connection = get_db_connection(AWS_SERVICES_DB_PATH) + cursor = connection.cursor() + result = cursor.execute(SELECT_QUERY_TEMPLATE.format(table_name=table_name)).fetchone() + + return row_type(*result) + + +def get_multiple_service_db_row(table_name): + """Return all rows from the given table name. + + Args: + table_name (str): Table name to search into. + + Yields: + Iterator[ServiceInspectorRow]: All the rows in the table. + """ + row_type = _get_service_row_type(table_name) + connection = get_db_connection(AWS_SERVICES_DB_PATH) + cursor = connection.cursor() + + for row in cursor.execute(SELECT_QUERY_TEMPLATE.format(table_name=table_name)): + yield row_type(*row) diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/event_monitor.py b/deps/wazuh_testing/wazuh_testing/modules/aws/event_monitor.py new file mode 100644 index 0000000000..757481ee78 --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/event_monitor.py @@ -0,0 +1,329 @@ +import re + +from wazuh_testing.modules.aws import VPC_FLOW_TYPE +from wazuh_testing.modules.aws.cli_utils import analyze_command_output + +PARSER_ERROR = r'.*wm_aws_read\(\): ERROR:.*' +MODULE_ERROR = r'.*wm_aws_run_s3\(\): ERROR: .*' +AWS_EVENT_HEADER = b'1:Wazuh-AWS:' + + +def make_aws_callback(pattern, prefix=''): + """Create a callback function from a text pattern. + + Args: + pattern (str): String to match on the log. + prefix (str): Regular expression used as prefix before the pattern. + + Returns: + lambda: Function that returns if there's a match in the file. + """ + pattern = r'\s+'.join(pattern.split()) + regex = re.compile(r'{}{}'.format(prefix, pattern)) + + return lambda line: regex.match(line) + + +def callback_detect_aws_module_called(parameters): + """Detect if aws module was called with correct parameters. + + Args: + parameters (list): Values to check. + + Returns: + Callable: Callback to match the line. + """ + regex = re.compile(fr'.*DEBUG: Launching S3 Command: {" ".join(parameters)}\n*') + return lambda line: regex.match(line) + + +def callback_detect_aws_error_for_missing_type(line): + """Detect if the AWS module displays an error about missing type. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it matches. + """ + + if re.match( + r".*ERROR: Undefined type for service.", line + ): + return line + + +def callback_detect_aws_legacy_module_warning(line): + """Detect if the AWS module displays a warning about legacy config. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it matches. + """ + + if re.match( + r".*WARNING: Deprecated config defined; please use current config definition at module 'aws-s3'.", line + ): + return line + + +def callback_detect_aws_module_warning(line): + """Detect if the AWS module displays a warning. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it matches. + """ + + if re.match(r".*WARNING: No buckets or services definitions found at module 'aws-s3'.", line): + return line + + +def callback_detect_aws_module_started(line): + """Detect if the AWS module was called. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it matches. + """ + + if re.match(r'.*DEBUG: Launching S3 Command: .*', line): + return line + + +def callback_detect_aws_empty_value(line): + """Detect if the AWS module displays a message about an empty value. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it matches. + """ + + if ( + re.match(r".*ERROR: Invalid \w+ type ''", line) or + re.match(r".*ERROR: Empty content for tag '\w+' at module 'aws-s3'.", line) or + re.match(r".*WARNING: Empty content for tag '\w+' at module 'aws-s3'.", line) + ): + return line + + +def callback_detect_aws_invalid_value(line): + """Detect if the AWS module displays a message about an invalid value. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it matches. + """ + + if ( + re.match(r".*ERROR: Invalid \w+ type '\w+'.*", line) or + re.match(r'.*ERROR: Invalid content for tag*', line) or + re.match(r'.*WARNING: Bucket: - Error parsing arguments.*', line), + re.match(r'.*WARNING: Service: - Error parsing arguments.*', line) + ): + return line + + +def callback_detect_bucket_or_service_call(line): + """Detect if bucket or service module was called. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it match. + """ + + if ( + re.match(r".*INFO: Executing Service Analysis:*", line) or + re.match(r'.*INFO: Executing Bucket Analysis:*', line) + ): + return line + + +def callback_detect_aws_module_start(line): + """Search for start message in the given line. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: Line if it match. + """ + + if re.match(r'.*INFO: Module AWS started*', line): + return line + + +def callback_detect_all_aws_err(line): + """Search for parse or module error message in the given line. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: line if it match. + """ + if re.match(PARSER_ERROR, line) or re.match(MODULE_ERROR, line): + return line + + +def callback_detect_aws_read_err(line): + """Search for parser error message in the given line. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: line if it match. + """ + if re.match(PARSER_ERROR, line): + return line + + +def callback_detect_aws_wmodule_err(line): + """Search for module error message in the given line. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: line if it match. + """ + if re.match(MODULE_ERROR, line): + return line + + +def callback_detect_event_processed(line): + """Search for event processed message in the given line. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: line if it match. + """ + if re.match(r'.*Found new log: .*', line): + return line + + +def callback_detect_event_processed_or_skipped(pattern): + """Search for event processed or skipped message in the given line. + + Args: + pattern (str): Pattern to match in line. + Returns: + Callable: Callback to match the given line. + """ + pattern_regex = re.compile(pattern) + return lambda line: pattern_regex.match(line) or callback_detect_event_processed(line) + + +def callback_detect_service_event_processed(expected_results, service_type): + if service_type == 'inspector': + regex = re.compile(fr"DEBUG: \+\+\+ {expected_results} events collected and processed in") + else: + regex = re.compile(fr'DEBUG: \+\+\+ Sent {expected_results} events to Analysisd') + return lambda line: regex.match(line) + + +def callback_event_sent_to_analysisd(line): + """Search for module header message in the given line. + + Args: + line (str): Line to match. + + Returns: + Optional[str]: line if it match. + """ + if line.startswith(AWS_EVENT_HEADER): + return line + + +def check_processed_logs_from_output(command_output, expected_results=1): + """Check for processed messages in the give output. + + Args: + command_output (str): Output to analyze. + expected_results (int, optional): Number of results to find. Defaults to 1. + """ + analyze_command_output( + command_output=command_output, + callback=callback_detect_event_processed, + expected_results=expected_results, + error_message='The AWS module did not process the expected number of events' + ) + + +def check_non_processed_logs_from_output(command_output, bucket_type, expected_results=1): + """Check for non processed messages in the give output. + + Args: + command_output (str): Output to analyze. + bucket_type (str): Bucket type to select the message. + expected_results (int, optional): Number of results to find. Defaults to 1. + """ + if bucket_type == VPC_FLOW_TYPE: + pattern = r'.*DEBUG: \+\+\+ No logs to process for .*' + else: + pattern = r'.*DEBUG: \+\+\+ No logs to process in bucket: ' + + analyze_command_output( + command_output, + callback=make_aws_callback(pattern), + expected_results=expected_results, + error_message='Some logs may were processed or the results found are more than expected' + ) + + +def check_marker_from_output(command_output, file_key, expected_results=1): + """Check for marker message in the given output. + + Args: + command_output (str): Output to analyze. + file_key (str): Value to check as a marker. + expected_results (int, optional): Number of results to find. Defaults to 1. + """ + pattern = fr".*DEBUG: \+\+\+ Marker: {file_key}" + + analyze_command_output( + command_output, + callback=make_aws_callback(pattern), + expected_results=expected_results, + error_message='The AWS module did not use the correct marker' + ) + + +def check_service_processed_logs_from_output( + command_output, events_sent, service_type, expected_results=1 +): + analyze_command_output( + command_output=command_output, + callback=callback_detect_service_event_processed(events_sent, service_type), + expected_results=expected_results, + error_message='The AWS module did not process the expected number of events' + ) + + +def check_service_non_processed_logs_from_output(command_output, service_type, expected_results=1): + if service_type == 'inspector': + pattern = r'DEBUG: \+\+\+ There are no new events in .*' + else: + pattern = r'DEBUG: \+\+\+ Sent \d+ events to Analysisd' + + analyze_command_output( + command_output, + callback=make_aws_callback(pattern), + expected_results=expected_results, + error_message='Some logs may were processed' + ) diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/exceptions.py b/deps/wazuh_testing/wazuh_testing/modules/aws/exceptions.py new file mode 100644 index 0000000000..a350d97621 --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/exceptions.py @@ -0,0 +1,2 @@ +class OutputAnalysisError(Exception): + pass diff --git a/deps/wazuh_testing/wazuh_testing/modules/aws/s3_utils.py b/deps/wazuh_testing/wazuh_testing/modules/aws/s3_utils.py new file mode 100644 index 0000000000..187707e5aa --- /dev/null +++ b/deps/wazuh_testing/wazuh_testing/modules/aws/s3_utils.py @@ -0,0 +1,89 @@ +"""AWS S3 related utils""" + +import gzip + +import boto3 +from botocore.exceptions import ClientError +from wazuh_testing import logger +from wazuh_testing.modules.aws.data_generator import get_data_generator + +session = boto3.Session(profile_name='qa') +s3 = session.resource('s3') + + +def upload_file(bucket_type, bucket_name): + """Upload a file to an S3 bucket. + + Args: + bucket_type (str): Bucket type to generate the data. + bucket_name (str): Bucket to upload. + + Returns: + str: The name of the file if was uploaded, else ''. + """ + dg = get_data_generator(bucket_type, bucket_name) + filename = dg.get_filename() + obj = s3.Object(bucket_name, filename) + + data = dg.get_data_sample().encode() if not dg.compress else gzip.compress(data=dg.get_data_sample().encode()) + + # Upload the file + try: + obj.put(Body=data) + except ClientError as e: + logger.error(e) + filename = '' + return filename + + +def delete_file(filename, bucket_name): + """Delete a given file from the bucket. + + Args: + filename (str): Full filename to delete. + bucket_name (str): Bucket that contains the file. + """ + s3.Object(bucket_name, filename).delete() + + +def file_exists(filename, bucket_name): + """Check if a file exists in a bucket. + + Args: + filename (str): Full filename to check. + bucket_name (str): Bucket that contains the file. + Returns: + bool: True if exists else False. + """ + exists = True + try: + s3.Object(bucket_name, filename).load() + except ClientError as error: + if error.response['Error']['Code'] == '404': + exists = False + + return exists + + +def get_last_file_key(bucket_type, bucket_name, execution_datetime): + """Return the last file key contained in a default path of a bucket. + + Args: + bucket_type (str): Bucket type to obtain the data generator. + bucket_name (str): Bucket that contains the file. + execution_datetime (datetime): Datetime to use to use as prefix. + + Returns: + str: The last key in the bucket. + """ + + dg = get_data_generator(bucket_type, bucket_name) + bucket = s3.Bucket(bucket_name) + last_key = None + + try: + *_, last_item = bucket.objects.filter(Prefix=dg.BASE_PATH or str(execution_datetime.year)) + last_key = last_item.key + except ValueError: + last_key = '' + return last_key diff --git a/deps/wazuh_testing/wazuh_testing/tools/utils.py b/deps/wazuh_testing/wazuh_testing/tools/utils.py index 346bfddb87..3a623f4648 100644 --- a/deps/wazuh_testing/wazuh_testing/tools/utils.py +++ b/deps/wazuh_testing/wazuh_testing/tools/utils.py @@ -115,6 +115,15 @@ def get_random_ip(): return fr"{randint(0,255)}.{randint(0,255)}.{randint(0,255)}.{randint(0,255)}" +def get_random_port() -> str: + """Create a port number. + + Return: + String: Random port number. + """ + return f"{randint(0, 10000)}" + + def get_random_string(string_length, digits=True): """Create a random string with specified length. @@ -161,13 +170,14 @@ def get_host_name(): def validate_interval_format(interval): """Validate that the interval passed has the format in which the last digit is a letter from those passed and - the other characters are between 0-9""" - if interval=='': + the other characters are between 0-9.""" + if interval == '': return False - if interval[-1] not in ['s','m', 'h','d','w','y'] or not isinstance(int(interval[0:-1]), numbers.Number): + if interval[-1] not in ['s', 'm', 'h', 'd', 'w', 'y'] or not isinstance(int(interval[0:-1]), numbers.Number): return False return True + def format_ipv6_long(ipv6_address): """Return the long form of the address representation in uppercase. diff --git a/requirements.txt b/requirements.txt index 336cfa1080..92f341ddb2 100644 --- a/requirements.txt +++ b/requirements.txt @@ -50,3 +50,4 @@ treelib==1.6.1 prettytable; platform_system == "Linux" mysql-connector-python==8.0.32; platform_system == 'Linux' and python_version >= '3.7' protobuf>=3.11.0,<=3.20.3; platform_system == 'Linux' and python_version >= '3.7' +boto3==1.26.11; platform_system == 'Linux' diff --git a/tests/integration/README.md b/tests/integration/README.md index d61a0a22ae..6a1d804772 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -154,6 +154,7 @@ Our newest integration tests are located in `wazuh-qa/tests/integration/`. They - _test_analysisd_ - _test_api_ - _test_authd_ +- _test_aws_ - _test_enrollment_ - _test_fim_ - _test_gcloud_ @@ -399,9 +400,9 @@ python3 -m pytest [options] [file_or_dir] [file_or_dir] [...] - `--wpk_version`: Specify the WPK package version used to upgrade on WPK tests. (ex. --wpk_version=v4.2.0). Note: This field is required to execute any WPK test and the WPK package must be previously created in the [repository](packages-dev.wazuh.com/trash/wpk/). -- `--wpk_package_path`: Specify the WPK package path used to upgrade on WPK tests. (ex. --wpk_package_path='packages-dev.wazuh.com/trash/wpk/'). +- `--wpk_package_path`: Specify the WPK package path used to upgrade on WPK tests. (ex. --wpk_package_path='packages-dev.wazuh.com/trash/wpk/'). This field is required to execute any WPK test. -- `--save-file`: Specify the files of execution of the tests that you need to download. (ex. --save-file=archives.json). +- `--save-file`: Specify the files of execution of the tests that you need to download. (ex. --save-file=archives.json). If the file does not exist while the test was executed, the test will run without problems but not save this file in the report. _Use `-h` to see the rest or check its [documentation](https://docs.pytest.org/en/latest/usage.html)._ diff --git a/tests/integration/test_aws/README.md b/tests/integration/test_aws/README.md new file mode 100644 index 0000000000..f06aac5e54 --- /dev/null +++ b/tests/integration/test_aws/README.md @@ -0,0 +1,180 @@ +# AWS Integration + +## Description + +It is a _wodle based_ module that has a capability to pull logs from several AWS services. + +## Tests directory structure + +```bash +wazuh-qa/tests/integration/test_aws +├── conftest.py +├── data +│   ├── configuration_template +│   │   ├── basic_test_module +│   │   ├── discard_regex_test_module +│   │   ├── only_logs_after_test_module +│   │   ├── path_suffix_test_module +│   │   ├── path_test_module +│   │   ├── regions_test_module +│   │   └── remove_from_bucket_test_module +│   └── test_cases +│   ├── basic_test_module +│   ├── discard_regex_test_module +│   ├── only_logs_after_test_module +│   ├── path_suffix_test_module +│   ├── path_test_module +│   ├── regions_test_module +│   └── remove_from_bucket_test_module +├── README.MD +├── test_basic.py +├── test_discard_regex.py +├── test_only_logs_after.py +├── test_path.py +├── test_path_suffix.py +├── test_regions.py +└── test_remove_from_bucket.py +``` + +## Deps directory structure + +```bash +wazuh-qa/deps/wazuh_testing/wazuh_testing/modules/aws +├── cli_utils.py +├── constants.py +├── data_generator.py +├── db_utils.py +├── event_monitor.py +├── __init__.py +└── s3_utils.py +``` + +## Requirements + +- The only extra dependency is `boto3` +- The module will assume there are already buckets, log groups and an inspector assessment with test data in AWS. + +## Configuration settings + +- **credentials** + Set the credentials at `$HOME/.aws/credentials` (being `HOME` the home directory of the user who runs the tests, more information [here](https://documentation.wazuh.com/current/amazon/services/prerequisites/credentials.html#profiles)) with the content: + +```ini +[qa] +aws_access_key_id = +aws_secret_access_key = +``` + +## Setting up a test environment + +You will need a proper environment to run the integration tests. You can use any virtual machine you wish. If you have +one already, go to the [integration tests section](#integration-tests) + +If you use [Vagrant](https://www.vagrantup.com/downloads.html) +or [VirtualBox](https://www.virtualbox.org/wiki/Downloads), it is important to install the `vbguest` plugin since some +tests modify the system date and there could be some synchronization issues. + +This guide will cover the following platforms: [Linux](#linux). + +You can run these tests on a manager or an agent. In case you are using an agent, please remember to register it and use +the correct version (Wazuh branch). + +_We are skipping Wazuh installation steps. For further information, +check [Wazuh documentation](https://documentation.wazuh.com/current/installation-guide/index.html)._ + +### Linux + +_We are using **Ubuntu 22.04** for this example:_ + +- Install **Wazuh** + +- Install python tests dependencies: + + ```shell script + # Install pip + apt install python3-pip + + # Clone your `wazuh-qa` repository within your testing environment + cd wazuh-qa + + # Install Python libraries + python3 -m pip install -r requirements.txt + + # Install test dependecies + python3 -m pip install deps/wazuh-testing + ``` + + +## Integration tests + +**DISCLAIMER:** this guide assumes you have a proper testing environment. If you do not, please check +our [testing environment guide](#setting-up-a-test-environment). + +### Pytest + +We use [pytest](https://docs.pytest.org/en/latest/contents.html) to run our integrity tests. Pytest will recursively +look for the closest `conftest` to import all the variables and fixtures needed for every test. If something is lacking +from the closest one, it will look for the next one (if possible) until reaching the current directory. This means we +need to run every test from the following path, where the general _conftest_ is: + +```shell script +cd wazuh-qa/tests/integration +``` + +To run any test, we just need to call `pytest` from `python3` using the following line: + +```shell script +python3 -m pytest [options] [file_or_dir] [file_or_dir] [...] +``` + +**Options:** + +- `v`: verbosity level (-v or -vv. Highly recommended to use -vv when tests are failing) +- `s`: shortcut for --capture=no. This will show the output in real time +- `x`: instantly exit after the first error. Very helpful when using a log truncate since it will keep the last failed + result +- `k`: only run tests which match the given substring expression (-k EXPRESSION) +- `m`: only run tests matching given expression (-m MARKEXPR) +- `--tier`: only run tests with given tier (ex. --tier 2) +- `--html`: generates a HTML report for the test results. (ex. --html=report.html) +- `--default-timeout`: overwrites the default timeout (in seconds). This value is used to make a test fail if a + condition is not met before the given time lapse. Some tests make use of this value and other has other fixed timeout + that cannot be modified. + +_Use `-h` to see the rest or check its [documentation](https://docs.pytest.org/en/latest/usage.html)._ + +Also, these integration tests are heavily based on [fixtures](https://docs.pytest.org/en/latest/fixture.html), so please +check its documentation for further information. + +#### AWS integration tests example + +```bash +# python3 -m pytest -vvx test_aws/ -k cloudtrail +=========================================================== test session starts ====================================================== +platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3 +cachedir: .pytest_cache +metadata: {'Python': '3.10.6', 'Platform': 'Linux-5.15.0-58-generic-x86_64-with-glibc2.35', +'Packages': {'pytest': '7.1.2', 'py': '1.10.0', 'pluggy': '1.0.0'}, +'Plugins': {'metadata': '2.0.2', 'html': '3.1.1', 'testinfra': '5.0.0'}} +rootdir: /home/vagrant/qa/tests/integration, configfile: pytest.ini +plugins: metadata-2.0.2, html-3.1.1, testinfra-5.0.0 +collected 15 items + +test_aws/test_basic.py::test_defaults[cloudtrail_defaults] PASSED [ 6%] +test_aws/test_discard_regex.py::test_discard_regex[cloudtrail_discard_regex] PASSED [ 13%] +test_aws/test_only_logs_after.py::test_without_only_logs_after[cloudtrail_without_only_logs_after] PASSED [ 20%] +test_aws/test_only_logs_after.py::test_with_only_logs_after[cloudtrail_with_only_logs_after] PASSED [ 26%] +test_aws/test_only_logs_after.py::test_multiple_calls[cloudtrail_only_logs_after_multiple_calls] PASSED [ 33%] +test_aws/test_path.py::test_path[cloudtrail_path_with_data] PASSED [ 40%] +test_aws/test_path.py::test_path[cloudtrail_path_without_data] PASSED [ 46%] +test_aws/test_path.py::test_path[cloudtrail_inexistent_path] PASSED [ 53%] +test_aws/test_path_suffix.py::test_path_suffix[cloudtrail_path_suffix_with_data] PASSED [ 60%] +test_aws/test_path_suffix.py::test_path_suffix[cloudtrail_path_suffix_without_data] PASSED [ 66%] +test_aws/test_path_suffix.py::test_path_suffix[cloudtrail_inexistent_path_suffix] PASSED [ 73%] +test_aws/test_regions.py::test_regions[cloudtrail_region_with_data] PASSED [ 80%] +test_aws/test_regions.py::test_regions[cloudtrail_regions_with_data] PASSED [ 86%] +test_aws/test_regions.py::test_regions[cloudtrail_inexistent_region] PASSED [ 93%] +test_aws/test_remove_from_bucket.py::test_remove_from_bucket[cloudtrail_remove_from_bucket] PASSED [100%] + +=============================================== 15 passed, 2 warnings in 332.67s (0:05:32) =========================================== +``` diff --git a/tests/integration/test_aws/conftest.py b/tests/integration/test_aws/conftest.py new file mode 100644 index 0000000000..88dec5e0b4 --- /dev/null +++ b/tests/integration/test_aws/conftest.py @@ -0,0 +1,161 @@ +import pytest +from wazuh_testing import logger +from wazuh_testing.modules.aws import ( + FAKE_CLOUDWATCH_LOG_GROUP, + PERMANENT_CLOUDWATCH_LOG_GROUP, +) +from wazuh_testing.modules.aws.cloudwatch_utils import ( + create_log_events, + create_log_group, + create_log_stream, + delete_log_group, + delete_log_stream, +) +from wazuh_testing.modules.aws.db_utils import delete_s3_db, delete_services_db +from wazuh_testing.modules.aws.s3_utils import delete_file, file_exists, upload_file +from wazuh_testing.tools.services import control_service + + +@pytest.fixture +def mark_cases_as_skipped(metadata): + if metadata['name'] in ['alb_remove_from_bucket', 'clb_remove_from_bucket', 'nlb_remove_from_bucket']: + pytest.skip(reason='ALB, CLB and NLB integrations are removing older logs from other region') + + +@pytest.fixture +def restart_wazuh_function_without_exception(daemon=None): + """Restart all Wazuh daemons.""" + try: + control_service("start", daemon=daemon) + except ValueError: + pass + + yield + + control_service('stop', daemon=daemon) + + +# S3 fixtures + +@pytest.fixture +def upload_and_delete_file_to_s3(metadata): + """Upload a file to S3 bucket and delete after the test ends. + + Args: + metadata (dict): Metadata to get the parameters. + """ + bucket_name = metadata['bucket_name'] + filename = upload_file(bucket_type=metadata['bucket_type'], bucket_name=metadata['bucket_name']) + if filename != '': + logger.debug('Uploaded file: %s to bucket "%s"', filename, bucket_name) + metadata['uploaded_file'] = filename + + yield + + if file_exists(filename=filename, bucket_name=bucket_name): + delete_file(filename=filename, bucket_name=bucket_name) + logger.debug('Deleted file: %s from bucket %s', filename, bucket_name) + + +@pytest.fixture +def delete_file_from_s3(metadata): + """Delete a file from S3 bucket after the test ends. + + Args: + metadata (dict): Metadata to get the parameters. + """ + yield + + bucket_name = metadata['bucket_name'] + filename = metadata.get('filename') + if filename is not None: + delete_file(filename=filename, bucket_name=bucket_name) + logger.debug('Deleted file: %s from bucket %s', filename, bucket_name) + + +# CloudWatch fixtures + +@pytest.fixture(name='create_log_stream') +def fixture_create_log_stream(metadata): + """Create a log stream with events and delete after the execution. + + Args: + metadata (dict): Metadata to get the parameters. + """ + SKIP_LOG_GROUP_CREATION = [PERMANENT_CLOUDWATCH_LOG_GROUP, FAKE_CLOUDWATCH_LOG_GROUP] + log_group_names = [item.strip() for item in metadata['log_group_name'].split(',')] + for log_group_name in log_group_names: + if log_group_name in SKIP_LOG_GROUP_CREATION: + continue + logger.debug('Creating log group: %s', log_group_name) + create_log_group(log_group_name) + log_stream = create_log_stream(log_group_name) + logger.debug('Created log stream "%s" within log group "%s"', log_stream, log_group_name) + create_log_events( + log_stream=log_stream, log_group=log_group_name, event_number=metadata.get('expected_results', 1) + ) + logger.debug('Created log events') + metadata['log_stream'] = log_stream + + yield + + for log_group_name in log_group_names: + if log_group_name in SKIP_LOG_GROUP_CREATION: + continue + delete_log_group(log_group_name) + logger.debug('Deleted log group: %s', log_group_name) + + +@pytest.fixture +def create_log_stream_in_existent_group(metadata): + """Create a log stream with events and delete after the execution. + + Args: + metadata (dict): Metadata to get the parameters. + """ + log_group_name = metadata['log_group_name'] + log_stream = create_log_stream(log_group_name) + logger.debug('Created log stream "%s" within log group "%s"', log_stream, log_group_name) + create_log_events(log_stream=log_stream, log_group=log_group_name) + logger.debug('Created log events') + metadata['log_stream'] = log_stream + + yield + + delete_log_stream(log_stream=log_stream, log_group=log_group_name) + logger.debug('Deleted log stream: %s', log_stream) + + +@pytest.fixture(name='delete_log_stream') +def fixture_delete_log_stream(metadata): + """Create a log stream with events and delete after the execution. + + Args: + metadata (dict): Metadata to get the parameters. + """ + yield + log_stream = metadata['log_stream'] + delete_log_stream(log_stream=log_stream) + logger.debug('Deleted log stream: %s', log_stream) + +# DB fixtures + + +@pytest.fixture +def clean_s3_cloudtrail_db(): + """Delete the DB file before and after the test execution""" + delete_s3_db() + + yield + + delete_s3_db() + + +@pytest.fixture +def clean_aws_services_db(): + """Delete the DB file before and after the test execution.""" + delete_services_db() + + yield + + delete_services_db() diff --git a/tests/integration/test_aws/data/configuration_template/basic_test_module/bucket_configuration_defaults.yaml b/tests/integration/test_aws/data/configuration_template/basic_test_module/bucket_configuration_defaults.yaml new file mode 100644 index 0000000000..507a734e36 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/basic_test_module/bucket_configuration_defaults.yaml @@ -0,0 +1,15 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME diff --git a/tests/integration/test_aws/data/configuration_template/basic_test_module/cloudwatch_configuration_defaults.yaml b/tests/integration/test_aws/data/configuration_template/basic_test_module/cloudwatch_configuration_defaults.yaml new file mode 100644 index 0000000000..6fc76e6537 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/basic_test_module/cloudwatch_configuration_defaults.yaml @@ -0,0 +1,17 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUP_NAME + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/configuration_template/basic_test_module/inspector_configuration_defaults.yaml b/tests/integration/test_aws/data/configuration_template/basic_test_module/inspector_configuration_defaults.yaml new file mode 100644 index 0000000000..2b6c6bd043 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/basic_test_module/inspector_configuration_defaults.yaml @@ -0,0 +1,15 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/configuration_template/discard_regex_test_module/configuration_discard_regex.yaml b/tests/integration/test_aws/data/configuration_template/discard_regex_test_module/configuration_discard_regex.yaml new file mode 100644 index 0000000000..cd7e6175e8 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/discard_regex_test_module/configuration_discard_regex.yaml @@ -0,0 +1,23 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - only_logs_after: + value: 2022-NOV-20 + - path: + value: PATH + - discard_regex: + attributes: + - field: DISCARD_FIELD + value: DISCARD_REGEX diff --git a/tests/integration/test_aws/data/configuration_template/log_groups_test_module/configuration_log_groups.yaml b/tests/integration/test_aws/data/configuration_template/log_groups_test_module/configuration_log_groups.yaml new file mode 100644 index 0000000000..17585a7f2d --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/log_groups_test_module/configuration_log_groups.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUP_NAME + - only_logs_after: + value: 2023-JAN-12 + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_with_only_logs_after.yaml b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_with_only_logs_after.yaml new file mode 100644 index 0000000000..1ccc03ee92 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_with_only_logs_after.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: "no" + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - only_logs_after: + value: ONLY_LOGS_AFTER + - path: + value: PATH diff --git a/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_without_only_logs_after.yaml b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_without_only_logs_after.yaml new file mode 100644 index 0000000000..0821bb0b8f --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/bucket_configuration_without_only_logs_after.yaml @@ -0,0 +1,17 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: "no" + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - path: + value: PATH diff --git a/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/cloudwatch_configuration_with_only_logs_after.yaml b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/cloudwatch_configuration_with_only_logs_after.yaml new file mode 100644 index 0000000000..5eacd69893 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/cloudwatch_configuration_with_only_logs_after.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: "no" + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUP_NAME + - only_logs_after: + value: ONLY_LOGS_AFTER + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/inspector_configuration_with_only_logs_after.yaml b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/inspector_configuration_with_only_logs_after.yaml new file mode 100644 index 0000000000..d88be0bb12 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/inspector_configuration_with_only_logs_after.yaml @@ -0,0 +1,17 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: "no" + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - only_logs_after: + value: ONLY_LOGS_AFTER + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/service_configuration_without_only_logs_after.yaml b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/service_configuration_without_only_logs_after.yaml new file mode 100644 index 0000000000..c16c07ec92 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/only_logs_after_test_module/service_configuration_without_only_logs_after.yaml @@ -0,0 +1,17 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: "no" + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUP_NAME + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_bucket_and_service_missing.yaml b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_bucket_and_service_missing.yaml new file mode 100644 index 0000000000..818a192d1a --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_bucket_and_service_missing.yaml @@ -0,0 +1,7 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' diff --git a/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_multiple_bucket_and_service_tags.yaml b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_multiple_bucket_and_service_tags.yaml new file mode 100644 index 0000000000..645fed742c --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_multiple_bucket_and_service_tags.yaml @@ -0,0 +1,47 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: cloudtrail + elements: + - aws_profile: + value: qa + - name: + value: wazuh-cloudtrail-integration-tests + - regions: + value: us-east-1 + - bucket: + attributes: + - type: cloudtrail + elements: + - aws_profile: + value: qa + - name: + value: wazuh-cloudtrail-integration-tests + - regions: + value: us-east-2 + - service: + attributes: + - type: cloudwatchlogs + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: wazuh-cloudwatchlogs-integration-tests + - regions: + value: us-east-1 + - service: + attributes: + - type: cloudwatchlogs + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: wazuh-cloudwatchlogs-integration-tests + - regions: + value: us-east-2 diff --git a/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_bucket.yaml b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_bucket.yaml new file mode 100644 index 0000000000..36cb9d2c6d --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_bucket.yaml @@ -0,0 +1,13 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + elements: + - aws_profile: + value: qa + - name: + value: wazuh-cloudtrail-integration-tests diff --git a/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_service.yaml b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_service.yaml new file mode 100644 index 0000000000..2a5e06849b --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_type_missing_in_service.yaml @@ -0,0 +1,13 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + elements: + - aws_profile: + value: qa + - name: + aws_log_groups: wazuh-cloudwatch-integration-tests diff --git a/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_bucket.yaml b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_bucket.yaml new file mode 100644 index 0000000000..5f433bbfa3 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_bucket.yaml @@ -0,0 +1,25 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - only_logs_after: + value: ONLY_LOGS_AFTER + - regions: + value: REGIONS + - path: + value: PATH + - path_suffix: + value: PATH_SUFFIX + - remove_from_bucket: + value: REMOVE_FROM_BUCKET diff --git a/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_service.yaml b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_service.yaml new file mode 100644 index 0000000000..1d8c91c7d3 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/parser_test_module/configuration_values_in_service.yaml @@ -0,0 +1,21 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUPS + - only_logs_after: + value: ONLY_LOGS_AFTER + - regions: + value: REGIONS + - remove_log_streams: + value: REMOVE_LOG_STREAMS diff --git a/tests/integration/test_aws/data/configuration_template/path_suffix_test_module/configuration_path_suffix.yaml b/tests/integration/test_aws/data/configuration_template/path_suffix_test_module/configuration_path_suffix.yaml new file mode 100644 index 0000000000..7b76a2dc22 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/path_suffix_test_module/configuration_path_suffix.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - only_logs_after: + value: 2022-NOV-20 + - path_suffix: + value: PATH_SUFFIX diff --git a/tests/integration/test_aws/data/configuration_template/path_test_module/configuration_path.yaml b/tests/integration/test_aws/data/configuration_template/path_test_module/configuration_path.yaml new file mode 100644 index 0000000000..62b5aaff26 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/path_test_module/configuration_path.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - only_logs_after: + value: 2022-NOV-20 + - path: + value: PATH diff --git a/tests/integration/test_aws/data/configuration_template/regions_test_module/bucket_configuration_regions.yaml b/tests/integration/test_aws/data/configuration_template/regions_test_module/bucket_configuration_regions.yaml new file mode 100644 index 0000000000..45b8fcb404 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/regions_test_module/bucket_configuration_regions.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - only_logs_after: + value: 2022-NOV-20 + - regions: + value: REGIONS diff --git a/tests/integration/test_aws/data/configuration_template/regions_test_module/cloudwatch_configuration_regions.yaml b/tests/integration/test_aws/data/configuration_template/regions_test_module/cloudwatch_configuration_regions.yaml new file mode 100644 index 0000000000..f6c2b339ac --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/regions_test_module/cloudwatch_configuration_regions.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUP_NAME + - only_logs_after: + value: 2023-JAN-12 + - regions: + value: REGIONS diff --git a/tests/integration/test_aws/data/configuration_template/regions_test_module/inspector_configuration_regions.yaml b/tests/integration/test_aws/data/configuration_template/regions_test_module/inspector_configuration_regions.yaml new file mode 100644 index 0000000000..ca344fac2e --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/regions_test_module/inspector_configuration_regions.yaml @@ -0,0 +1,17 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - only_logs_after: + value: 2023-JAN-12 + - regions: + value: REGIONS diff --git a/tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_from_bucket.yaml b/tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_from_bucket.yaml new file mode 100644 index 0000000000..5c8d3f24a7 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_from_bucket.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - bucket: + attributes: + - type: BUCKET_TYPE + elements: + - aws_profile: + value: qa + - name: + value: BUCKET_NAME + - remove_from_bucket: + value: 'yes' + - path: + value: PATH diff --git a/tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_log_stream.yaml b/tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_log_stream.yaml new file mode 100644 index 0000000000..2fbdbf0737 --- /dev/null +++ b/tests/integration/test_aws/data/configuration_template/remove_from_bucket_test_module/configuration_remove_log_stream.yaml @@ -0,0 +1,19 @@ +- sections: + - section: wodle + attributes: + - name: aws-s3 + elements: + - disabled: + value: 'no' + - service: + attributes: + - type: SERVICE_TYPE + elements: + - aws_profile: + value: qa + - aws_log_groups: + value: LOG_GROUP_NAME + - remove_log_streams: + value: 'yes' + - regions: + value: us-east-1 diff --git a/tests/integration/test_aws/data/test_cases/basic_test_module/cases_bucket_defaults.yaml b/tests/integration/test_aws/data/test_cases/basic_test_module/cases_bucket_defaults.yaml new file mode 100644 index 0000000000..cedf6c83b2 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/basic_test_module/cases_bucket_defaults.yaml @@ -0,0 +1,125 @@ +- name: cloudtrail_defaults + description: CloudTrail default configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + +- name: vpc_defaults + description: VPC default configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + +- name: config_defaults + description: Config default configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + +- name: alb_defaults + description: ALB default configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + +- name: clb_defaults + description: CLB default configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + +- name: nlb_defaults + description: NLB default configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + +- name: kms_defaults + description: KMS default configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + +- name: macie_defaults + description: CloudTrail default configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + +- name: trusted_advisor_defaults + description: Trusted Advisor default configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + +- name: guardduty_defaults + description: GuardDuty default configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + +- name: native_guardduty_defaults + description: Native GuardDuty default configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + +- name: waf_defaults + description: WAF default configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + +- name: server_access_defaults + description: Server Access default configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + +- name: cisco_umbrella_defaults + description: Umbrella default configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests diff --git a/tests/integration/test_aws/data/test_cases/basic_test_module/cases_cloudwatch_defaults.yaml b/tests/integration/test_aws/data/test_cases/basic_test_module/cases_cloudwatch_defaults.yaml new file mode 100644 index 0000000000..ca0c2e7ada --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/basic_test_module/cases_cloudwatch_defaults.yaml @@ -0,0 +1,8 @@ +- name: cloudwatchlogs_defaults + description: CloudWatch default configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests diff --git a/tests/integration/test_aws/data/test_cases/basic_test_module/cases_inspector_defaults.yaml b/tests/integration/test_aws/data/test_cases/basic_test_module/cases_inspector_defaults.yaml new file mode 100644 index 0000000000..094da2a539 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/basic_test_module/cases_inspector_defaults.yaml @@ -0,0 +1,6 @@ +- name: inspector_defaults + description: Inspector default configurations + configuration_parameters: + SERVICE_TYPE: inspector + metadata: + service_type: inspector diff --git a/tests/integration/test_aws/data/test_cases/discard_regex_test_module/cases_discard_regex.yaml b/tests/integration/test_aws/data/test_cases/discard_regex_test_module/cases_discard_regex.yaml new file mode 100644 index 0000000000..7c37319fb6 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/discard_regex_test_module/cases_discard_regex.yaml @@ -0,0 +1,238 @@ +- name: cloudtrail_discard_regex + description: CloudTrail discard regex configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + DISCARD_FIELD: eventSource + DISCARD_REGEX: .*ec2.amazonaws.com.* + PATH: '' + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: eventSource + discard_regex: .*ec2.amazonaws.com.* + found_logs: 5 + skipped_logs: 1 + +- name: vpc_discard_regex + description: VPC discard regex configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + DISCARD_FIELD: srcport + DISCARD_REGEX: "5319" + PATH: '' + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: srcport + discard_regex: "5319" + found_logs: 5 + skipped_logs: 1 + +- name: config_discard_regex + description: Config discard regex configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + DISCARD_FIELD: configuration.complianceType + DISCARD_REGEX: .*COMPLIANT.* + PATH: '' + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: configuration.complianceType + discard_regex: .*COMPLIANT.* + found_logs: 5 + skipped_logs: 1 + +- name: alb_discard_regex + description: ALB discard regex configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + DISCARD_FIELD: elb_status_code + DISCARD_REGEX: '401' + PATH: '' + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: elb_status_code + discard_regex: '401' + found_logs: 5 + skipped_logs: 1 + +- name: clb_discard_regex + description: CLB discard regex configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + DISCARD_FIELD: elb_status_code + DISCARD_REGEX: '401' + PATH: '' + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: elb_status_code + discard_regex: '401' + found_logs: 5 + skipped_logs: 1 + +- name: nlb_discard_regex + description: NLB discard regex configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + DISCARD_FIELD: listener + DISCARD_REGEX: 0CMK2UAG108C7AXK + PATH: '' + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: listener + discard_regex: 0CMK2UAG108C7AXK + found_logs: 5 + skipped_logs: 1 + +- name: kms_discard_regex + description: KMS discard regex configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + DISCARD_FIELD: eventName + DISCARD_REGEX: MatchDataKey + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: eventName + discard_regex: MatchDataKey + found_logs: 3 + skipped_logs: 1 + +- name: macie_discard_regex + description: Macie discard regex configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + DISCARD_FIELD: severity + DISCARD_REGEX: LOW + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: severity + discard_regex: LOW + found_logs: 3 + skipped_logs: 1 + +- name: trusted_advisor_discard_regex + description: Trusted Advisor discard regex configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + DISCARD_FIELD: status + DISCARD_REGEX: ERROR + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: status + discard_regex: ERROR + found_logs: 3 + skipped_logs: 1 + +- name: guardduty_discard_regex + description: GuardDuty discard regex configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + DISCARD_FIELD: partition + DISCARD_REGEX: aws-test + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: partition + discard_regex: aws-test + found_logs: 3 + skipped_logs: 1 + +- name: native_guardduty_discard_regex + description: Native GuardDuty discard regex configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + DISCARD_FIELD: partition + DISCARD_REGEX: aws-test + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: partition + discard_regex: aws-test + found_logs: 3 + skipped_logs: 1 + +- name: waf_discard_regex + description: WAF discard regex configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + DISCARD_FIELD: action + DISCARD_REGEX: ALLOW + PATH: '' + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: action + discard_regex: ALLOW + found_logs: 3 + skipped_logs: 1 + +- name: server_access_discard_regex + description: Server Access discard regex configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + DISCARD_FIELD: http_status + DISCARD_REGEX: '200' + PATH: '' + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: http_status + discard_regex: '200' + found_logs: 3 + skipped_logs: 1 + +- name: cisco_umbrella_discard_regex + description: CloudTrail discard regex configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + DISCARD_FIELD: action + DISCARD_REGEX: Blocked + PATH: dnslogs + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + only_logs_after: 2022-NOV-20 + discard_field: action + discard_regex: Blocked + found_logs: 3 + skipped_logs: 1 + path: dnslogs diff --git a/tests/integration/test_aws/data/test_cases/log_groups_test_module/cases_log_groups.yaml b/tests/integration/test_aws/data/test_cases/log_groups_test_module/cases_log_groups.yaml new file mode 100644 index 0000000000..aefd5f59f7 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/log_groups_test_module/cases_log_groups.yaml @@ -0,0 +1,21 @@ +- name: cloudwatchlogs_log_groups_with_data + description: CloudWatch log groups configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests,temporary-log-group + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests,temporary-log-group + only_logs_after: 2023-JAN-12 + expected_results: 3 + +- name: cloudwatchlogs_inexistent_log_group + description: CloudWatch log group configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: fake-log-group + metadata: + service_type: cloudwatchlogs + log_group_name: fake-log-group + only_logs_after: 2023-JAN-12 + expected_results: 0 diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_multiple_calls.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_multiple_calls.yaml new file mode 100644 index 0000000000..12b19b6226 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_multiple_calls.yaml @@ -0,0 +1,98 @@ +- name: cloudtrail_only_logs_after_multiple_calls + description: CloudTrail only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + +- name: vpc_only_logs_after_multiple_calls + description: VPC only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + +- name: config_only_logs_after_multiple_calls + description: Config only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + +- name: alb_only_logs_after_multiple_calls + description: ALB only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + +- name: clb_only_logs_after_multiple_calls + description: CLB only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + +- name: nlb_only_logs_after_multiple_calls + description: NLB only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + +- name: kms_only_logs_after_multiple_calls + description: KMS only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + +- name: macie_only_logs_after_multiple_calls + description: Macie only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + +- name: trusted_advisor_only_logs_after_multiple_calls + description: Trusted Advisor only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + +- name: guardduty_only_logs_after_multiple_calls + description: GuardDuty only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + +- name: native_guardduty_only_logs_after_multiple_calls + description: Native GuardDuty only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + +- name: waf_only_logs_after_multiple_calls + description: WAF only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + +- name: server_access_only_logs_after_multiple_calls + description: Server Access only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + +- name: cisco_umbrella_only_logs_after_multiple_calls + description: Umbrella only_logs_after multiple calls configurations + configuration_parameters: + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + path: dnslogs diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_with_only_logs_after.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_with_only_logs_after.yaml new file mode 100644 index 0000000000..a466dcbf44 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_with_only_logs_after.yaml @@ -0,0 +1,183 @@ +- name: cloudtrail_with_only_logs_after + description: CloudTrail only logs after configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 5 + +- name: vpc_with_only_logs_after + description: VPC only logs after configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: config_with_only_logs_after + description: Config only logs after configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 5 + +- name: alb_with_only_logs_after + description: ALB only logs after configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 5 + +- name: clb_with_only_logs_after + description: CLB only logs after configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 5 + +- name: nlb_with_only_logs_after + description: NLB only logs after configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 5 + +- name: kms_with_only_logs_after + description: KMS only logs after configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: macie_with_only_logs_after + description: Macie only logs after configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: trusted_avisor_with_only_logs_after + description: Trusted Advisor only logs after configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: guardduty_with_only_logs_after + description: GuardDuty only logs after configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: native_guardduty_with_only_logs_after + description: Native GuardDuty only logs after configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: waf_with_only_logs_after + description: WAF only logs after configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + +- name: server_access_with_only_logs_after + description: Server Access only logs after configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: '' + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + table_name: s3_server_access + +- name: cisco_umbrella_with_only_logs_after + description: Umbrella only logs after configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + PATH: dnslogs + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + path: dnslogs diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_without_only_logs_after.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_without_only_logs_after.yaml new file mode 100644 index 0000000000..8b622f44f8 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_bucket_without_only_logs_after.yaml @@ -0,0 +1,155 @@ +- name: cloudtrail_without_only_logs_after + description: CloudTrail only logs after configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH: '' + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + expected_results: 1 + +- name: vpc_without_only_logs_after + description: VPC only logs after configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH: '' + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + expected_results: 1 + +- name: config_without_only_logs_after + description: Config only logs after configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH: '' + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + expected_results: 1 + +- name: alb_without_only_logs_after + description: ALB only logs after configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + PATH: '' + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + expected_results: 1 + +- name: clb_without_only_logs_after + description: CLB only logs after configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + PATH: '' + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + expected_results: 1 + +- name: nlb_without_only_logs_after + description: NLB only logs after configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + PATH: '' + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + expected_results: 1 + +- name: kms_without_only_logs_after + description: KMS only logs after configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + expected_results: 1 + +- name: macie_without_only_logs_after + description: Macie only logs after configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + expected_results: 1 + +- name: trusted_advisor_without_only_logs_after + description: Trusted Advisor only logs after configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + expected_results: 1 + +- name: guardduty_without_only_logs_after + description: GuardDuty only logs after configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + expected_results: 1 + +- name: native_guardduty_without_only_logs_after + description: Native GuardDuty only logs after configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + expected_results: 1 + +- name: waf_without_only_logs_after + description: WAF only logs after configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + PATH: '' + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + expected_results: 1 + +- name: server_access_without_only_logs_after + description: Server Access only logs after configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + PATH: '' + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + expected_results: 1 + table_name: s3_server_access + +- name: cisco_umbrella_without_only_logs_after + description: Umbrella only logs after configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + PATH: dnslogs + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + expected_results: 1 + path: dnslogs diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_multiple_calls.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_multiple_calls.yaml new file mode 100644 index 0000000000..808e86d511 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_multiple_calls.yaml @@ -0,0 +1,6 @@ +- name: cloudwatchlogs_only_logs_after_multiple_calls + description: CloudWatch only_logs_after multiple calls configurations + configuration_parameters: + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_with_only_logs_after.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_with_only_logs_after.yaml new file mode 100644 index 0000000000..6fc8afb157 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_cloudwatch_with_only_logs_after.yaml @@ -0,0 +1,12 @@ +- name: cloudwatchlogs_with_only_logs_after + description: CloudWatch only logs after configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests + ONLY_LOGS_AFTER: 2022-NOV-20 + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests + only_logs_after: 2022-NOV-20 + expected_results: 3 + log_stream: permanent-logs diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_multiple_calls.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_multiple_calls.yaml new file mode 100644 index 0000000000..738095e861 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_multiple_calls.yaml @@ -0,0 +1,5 @@ +- name: inspector_only_logs_after_multiple_calls + description: Inspector only_logs_after multiple calls configurations + configuration_parameters: + metadata: + service_type: inspector diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_with_only_logs_after.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_with_only_logs_after.yaml new file mode 100644 index 0000000000..860c37e797 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_inspector_with_only_logs_after.yaml @@ -0,0 +1,9 @@ +- name: inspector_with_only_logs_after + description: Inspector only logs after configurations + configuration_parameters: + SERVICE_TYPE: inspector + ONLY_LOGS_AFTER: 2023-JAN-30 + metadata: + service_type: inspector + only_logs_after: 2023-JAN-30 + expected_results: 4 diff --git a/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_service_without_only_logs_after.yaml b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_service_without_only_logs_after.yaml new file mode 100644 index 0000000000..40414b43d6 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/only_logs_after_test_module/cases_service_without_only_logs_after.yaml @@ -0,0 +1,9 @@ +- name: cloudwatchlogs_without_only_logs_after + description: CloudWatch only logs after configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests + expected_results: 1 diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_bucket_and_service_missing.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_bucket_and_service_missing.yaml new file mode 100644 index 0000000000..a1b300a75a --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_bucket_and_service_missing.yaml @@ -0,0 +1,4 @@ +- name: parser_bucket_and_service_missing + description: Parser bucket and service missing configurations + configuration_parameters: [] + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_bucket.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_bucket.yaml new file mode 100644 index 0000000000..f230042ece --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_bucket.yaml @@ -0,0 +1,71 @@ +- name: parser_empty_type_in_bucket + description: Parser empty type in bucket + configuration_parameters: + BUCKET_TYPE: '' + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_name_in_bucket + description: Parser empty name in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: '' + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_only_logs_after_in_bucket + description: Parser empty only_logs_after in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: '' + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_regions_in_bucket + description: Parser empty regions in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: '' + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_path_in_bucket + description: Parser empty path in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: '' + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_path_suffix_in_bucket + description: Parser empty path_suffix in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: '' + REMOVE_FROM_BUCKET: 'no' + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_service.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_service.yaml new file mode 100644 index 0000000000..d252f271d0 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_empty_values_in_service.yaml @@ -0,0 +1,39 @@ +- name: parser_empty_type_in_service + description: Parser empty type in service + configuration_parameters: + SERVICE_TYPE: '' + LOG_GROUPS: wazuh-cloudwatch-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_log_groups_in_service + description: Parser empty log_groups in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: '' + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_only_logs_after_in_service + description: Parser empty only_logs_after in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: wazuh-cloudwatchlogs-integration-tests + ONLY_LOGS_AFTER: '' + REGIONS: us-east-1 + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_empty_regions_in_service + description: Parser empty regions in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: wazuh-cloudwatchlogs-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: '' + REMOVE_FROM_BUCKET: 'no' + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_bucket.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_bucket.yaml new file mode 100644 index 0000000000..9edda13e2a --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_bucket.yaml @@ -0,0 +1,83 @@ +- name: parser_invalid_type_in_bucket + description: Parser invalid type in bucket + configuration_parameters: + BUCKET_TYPE: invalid_value + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_invalid_name_in_bucket + description: Parser invalid name in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: 1 + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_invalid_only_logs_after_in_bucket + description: Parser invalid only_logs_after in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_invalid_regions_in_bucket + description: Parser invalid regions in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: 1 + PATH: test_prefix + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_invalid_path_in_bucket + description: Parser invalid path in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test-prefix> + PATH_SUFFIX: test_suffix + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_invalid_path_suffix_in_bucket + description: Parser invalid path_suffix in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test-suffix> + REMOVE_FROM_BUCKET: 'no' + metadata: [] + +- name: parser_invalid_remove_from_bucket_in_bucket + description: Parser invalid remove_from_bucket in bucket + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + PATH: test_prefix + PATH_SUFFIX: test-suffix + REMOVE_FROM_BUCKET: nein + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_service.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_service.yaml new file mode 100644 index 0000000000..fb739358fc --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_invalid_values_in_service.yaml @@ -0,0 +1,49 @@ +- name: parser_invalid_type_in_service + description: Parser invalid type in service + configuration_parameters: + SERVICE_TYPE: fakeservice + LOG_GROUPS: wazuh-cloudwatch-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + REMOVE_LOG_STREAMS: 'no' + metadata: [] + +- name: parser_invalid_log_groups_in_service + description: Parser invalid log_groups in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: invalid_log_group> + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: us-east-1 + REMOVE_LOG_STREAMS: 'no' + metadata: [] + +- name: parser_invalid_only_logs_after_in_service + description: Parser invalid only_logs_after in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: wazuh-cloudwatchlogs-integration-tests + ONLY_LOGS_AFTER: JAN-31 + REGIONS: us-east-1 + REMOVE_LOG_STREAMS: 'no' + metadata: [] + +- name: parser_invalid_regions_in_service + description: Parser invalid regions in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: wazuh-cloudwatchlogs-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: 1 + REMOVE_LOG_STREAMS: 'no' + metadata: [] + +- name: parser_invalid_remove_log_stream_in_service + description: Parser invalid remove_log_stream in service + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUPS: wazuh-cloudwatchlogs-integration-tests + ONLY_LOGS_AFTER: 2023-JAN-31 + REGIONS: 1 + REMOVE_LOG_STREAMS: nein + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_multiple_bucket_and_service_tags.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_multiple_bucket_and_service_tags.yaml new file mode 100644 index 0000000000..6b07c332e6 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_multiple_bucket_and_service_tags.yaml @@ -0,0 +1,4 @@ +- name: parser_mutiple_bucket_and_service_tags + description: Parser multiple bucket and service tags configurations + configuration_parameters: [] + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_bucket.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_bucket.yaml new file mode 100644 index 0000000000..44cef7e1dd --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_bucket.yaml @@ -0,0 +1,4 @@ +- name: parser_type_missing_in_bucket + description: Parser type missing in bucket configurations + configuration_parameters: [] + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_service.yaml b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_service.yaml new file mode 100644 index 0000000000..d8ba3d2b20 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/parser_test_module/cases_type_missing_in_service.yaml @@ -0,0 +1,4 @@ +- name: parser_type_missing_in_service + description: Parser type missing in service configurations + configuration_parameters: [] + metadata: [] diff --git a/tests/integration/test_aws/data/test_cases/path_suffix_test_module/cases_path_suffix.yaml b/tests/integration/test_aws/data/test_cases/path_suffix_test_module/cases_path_suffix.yaml new file mode 100644 index 0000000000..b95f1695a6 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/path_suffix_test_module/cases_path_suffix.yaml @@ -0,0 +1,116 @@ +- name: cloudtrail_path_suffix_with_data + description: CloudTrail path_suffix configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH_SUFFIX: test_suffix + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: test_suffix + expected_results: 1 + +- name: cloudtrail_path_suffix_without_data + description: CloudTrail path_suffix configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH_SUFFIX: empty_suffix + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: empty_suffix + expected_results: 0 + +- name: cloudtrail_inexistent_path_suffix + description: CloudTrail path_suffix configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH_SUFFIX: inexistent_suffix + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: inexistent_suffix + expected_results: 0 + +- name: vpc_path_suffix_with_data + description: VPC path_suffix configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH_SUFFIX: test_suffix + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: test_suffix + expected_results: 1 + +- name: config_path_suffix_with_data + description: Config path_suffix configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH_SUFFIX: test_suffix + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: test_suffix + expected_results: 1 + +- name: vpc_path_suffix_without_data + description: VPC path_suffix configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH_SUFFIX: empty_suffix + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: empty_suffix + expected_results: 0 + +- name: config_path_suffix_without_data + description: Config path_suffix configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH_SUFFIX: empty_suffix + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: empty_suffix + expected_results: 0 + +- name: vpc_inexistent_path_suffix + description: VPC path_suffix configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH_SUFFIX: inexistent_suffix + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: inexistent_suffix + expected_results: 0 + +- name: config_inexistent_path_suffix + description: Config path_suffix configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH_SUFFIX: inexistent_suffix + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + path_suffix: inexistent_suffix + expected_results: 0 diff --git a/tests/integration/test_aws/data/test_cases/path_test_module/cases_path.yaml b/tests/integration/test_aws/data/test_cases/path_test_module/cases_path.yaml new file mode 100644 index 0000000000..86b04ea2aa --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/path_test_module/cases_path.yaml @@ -0,0 +1,548 @@ +- name: cloudtrail_path_with_data + description: CloudTrail path configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH: test_prefix + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: cloudtrail_path_without_data + description: CloudTrail path configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH: empty_prefix + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: cloudtrail_inexistent_path + description: CloudTrail path configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: vpc_path_with_data + description: VPC path configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH: test_prefix + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: vpc_path_without_data + description: VPC path configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH: empty_prefix + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: cisco_umbrella_path_with_data + description: Umbrella path configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + PATH: test_prefix/dnslogs + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix/dnslogs + expected_results: 1 + +- name: cisco_umbrella_path_without_data + description: Umbrella path configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + PATH: empty_prefix + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: vpc_inexistent_path + description: CloudTrail path configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: config_path_with_data + description: Config path configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH: test_prefix + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: config_path_without_data + description: Config path configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH: empty_prefix + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: config_inexistent_path + description: Config path configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: alb_path_with_data + description: ALB path configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + PATH: test_prefix + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: alb_path_without_data + description: ALB path configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + PATH: empty_prefix + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: alb_inexistent_path + description: ALB path configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: clb_path_with_data + description: CLB path configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + PATH: test_prefix + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: clb_path_without_data + description: CLB path configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + PATH: empty_prefix + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: clb_inexistent_path + description: CLB path configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: nlb_path_with_data + description: NLB path configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + PATH: test_prefix + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: nlb_path_without_data + description: NLB path configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + PATH: empty_prefix + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: nlb_inexistent_path + description: NLB path configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: kms_path_with_data + description: KMS path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + PATH: test_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: kms_path_without_data + description: KMS path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + PATH: empty_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: kms_inexistent_path + description: KMS path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: macie_path_with_data + description: Macie path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + PATH: test_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: macie_path_without_data + description: Macie path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + PATH: empty_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: macie_inexistent_path + description: Macie path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: trusted_advisor_path_with_data + description: Trusted Advisor path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + PATH: test_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: trusted_advisor_path_without_data + description: Trusted Advisor path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + PATH: empty_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: trusted_advisor_inexistent_path + description: Trusted Advisor path configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: guardduty_path_with_data + description: GuardDuty path configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + PATH: test_prefix + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: guardduty_path_without_data + description: GuardDuty path configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + PATH: empty_prefix + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: guardduty_inexistent_path + description: GuardDuty path configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: native_guardduty_path_with_data + description: Native GuardDuty path configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + PATH: test_prefix + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: native_guardduty_path_without_data + description: Native GuardDuty path configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + PATH: empty_prefix + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: native_guardduty_inexistent_path + description: Native GuardDuty path configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: waf_path_with_data + description: WAF path configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + PATH: test_prefix + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + +- name: waf_path_without_data + description: WAF path configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + PATH: empty_prefix + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + +- name: waf_inexistent_path + description: WAF path configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + +- name: server_access_path_with_data + description: Server Access path configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + PATH: test_prefix + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + only_logs_after: 2022-NOV-20 + path: test_prefix + expected_results: 1 + table_name: s3_server_access + +- name: server_access_path_without_data + description: Server Access path configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + PATH: empty_prefix + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + only_logs_after: 2022-NOV-20 + path: empty_prefix + expected_results: 0 + table_name: s3_server_access + +- name: server_access_inexistent_path + description: Server Access path configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 + table_name: s3_server_access + +- name: cisco_umbrella_inexistent_path + description: Umbrella path configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + PATH: inexistent_prefix + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + only_logs_after: 2022-NOV-20 + path: inexistent_prefix + expected_results: 0 diff --git a/tests/integration/test_aws/data/test_cases/regions_test_module/cases_bucket_regions.yaml b/tests/integration/test_aws/data/test_cases/regions_test_module/cases_bucket_regions.yaml new file mode 100644 index 0000000000..07231f255a --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/regions_test_module/cases_bucket_regions.yaml @@ -0,0 +1,233 @@ +- name: cloudtrail_region_with_data + description: CloudTrail regions configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + REGIONS: us-east-1 + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1 + expected_results: 3 + +- name: cloudtrail_regions_with_data + description: CloudTrail regions configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1,us-east-2 + expected_results: 5 + +- name: cloudtrail_inexistent_region + description: CloudTrail regions configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + REGIONS: us-fake-1 + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-fake-1 + expected_results: 0 + +- name: vpc_region_with_data + description: VPC regions configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + REGIONS: us-east-1 + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1 + expected_results: 3 + +- name: config_region_with_data + description: Config regions configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + REGIONS: us-east-1 + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1 + expected_results: 3 + +- name: alb_region_with_data + description: ALB regions configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + REGIONS: us-east-1 + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1 + expected_results: 3 + +- name: vpc_regions_with_data + description: VPC regions configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1,us-east-2 + expected_results: 5 + +- name: config_regions_with_data + description: Config regions configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1,us-east-2 + expected_results: 5 + +- name: alb_regions_with_data + description: ALB regions configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1,us-east-2 + expected_results: 5 + +- name: vpc_inexistent_region + description: VPC regions configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + REGIONS: us-fake-1 + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-fake-1 + expected_results: 0 + +- name: config_inexistent_region + description: Config regions configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + REGIONS: us-fake-1 + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-fake-1 + expected_results: 0 + +- name: alb_inexistent_region + description: ALB regions configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + REGIONS: us-fake-1 + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-fake-1 + expected_results: 0 + +- name: clb_region_with_data + description: CLB regions configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + REGIONS: us-east-1 + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1 + expected_results: 3 + +- name: clb_regions_with_data + description: CLB regions configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1,us-east-2 + expected_results: 5 + +- name: clb_inexistent_region + description: CLB regions configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + REGIONS: us-fake-1 + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-fake-1 + expected_results: 0 + +- name: nlb_region_with_data + description: NLB regions configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + REGIONS: us-east-1 + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1 + expected_results: 3 + +- name: nlb_regions_with_data + description: NLB regions configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-east-1,us-east-2 + expected_results: 5 + +- name: nlb_inexistent_region + description: NLB regions configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + REGIONS: us-fake-1 + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + only_logs_after: 2022-NOV-20 + regions: us-fake-1 + expected_results: 0 diff --git a/tests/integration/test_aws/data/test_cases/regions_test_module/cases_cloudwatch_regions.yaml b/tests/integration/test_aws/data/test_cases/regions_test_module/cases_cloudwatch_regions.yaml new file mode 100644 index 0000000000..d5f4c3b8e4 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/regions_test_module/cases_cloudwatch_regions.yaml @@ -0,0 +1,38 @@ +- name: cloudwatchlogs_region_with_data + description: CloudWatch regions configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests + REGIONS: us-east-1 + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests + only_logs_after: 2023-JAN-12 + regions: us-east-1 + expected_results: 3 + +- name: cloudwatchlogs_regions_with_data + description: CloudWatch regions configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests + REGIONS: us-east-1,us-east-2 + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests + only_logs_after: 2023-JAN-12 + regions: us-east-1,us-east-2 + expected_results: 3 + +- name: cloudwatchlogs_inexistent_region + description: CloudWatch regions configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: wazuh-cloudwatchlogs-integration-tests + REGIONS: us-fake-1 + metadata: + service_type: cloudwatchlogs + log_group_name: wazuh-cloudwatchlogs-integration-tests + only_logs_after: 2023-JAN-12 + regions: us-fake-1 + expected_results: 0 diff --git a/tests/integration/test_aws/data/test_cases/regions_test_module/cases_inspector_regions.yaml b/tests/integration/test_aws/data/test_cases/regions_test_module/cases_inspector_regions.yaml new file mode 100644 index 0000000000..fc1fbbd39b --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/regions_test_module/cases_inspector_regions.yaml @@ -0,0 +1,32 @@ +- name: inspector_region_with_data + description: Inspector regions configurations + configuration_parameters: + SERVICE_TYPE: inspector + REGIONS: us-east-1 + metadata: + service_type: inspector + only_logs_after: 2023-JAN-12 + regions: us-east-1 + expected_results: 4 + +- name: inspector_regions_with_data + description: Inspector regions configurations + configuration_parameters: + SERVICE_TYPE: inspector + REGIONS: us-east-1,us-east-2 + metadata: + service_type: inspector + only_logs_after: 2023-JAN-12 + regions: us-east-1,us-east-2 + expected_results: 4 + +- name: inspector_inexistent_region + description: Inspector regions configurations + configuration_parameters: + SERVICE_TYPE: inspector + REGIONS: us-fake-1 + metadata: + service_type: inspector + only_logs_after: 2023-JAN-12 + regions: us-fake-1 + expected_results: 0 diff --git a/tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_from_bucket.yaml b/tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_from_bucket.yaml new file mode 100644 index 0000000000..eb26143283 --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_from_bucket.yaml @@ -0,0 +1,140 @@ +- name: cloudtrail_remove_from_bucket + description: CloudTrail remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: cloudtrail + BUCKET_NAME: wazuh-cloudtrail-integration-tests + PATH: '' + metadata: + bucket_type: cloudtrail + bucket_name: wazuh-cloudtrail-integration-tests + +- name: vpc_remove_from_bucket + description: VPC remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: vpcflow + BUCKET_NAME: wazuh-vpcflow-integration-tests + PATH: '' + metadata: + bucket_type: vpcflow + bucket_name: wazuh-vpcflow-integration-tests + +- name: config_remove_from_bucket + description: Config remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: config + BUCKET_NAME: wazuh-config-integration-tests + PATH: '' + metadata: + bucket_type: config + bucket_name: wazuh-config-integration-tests + +- name: alb_remove_from_bucket + description: ALB remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: alb + BUCKET_NAME: wazuh-alb-integration-tests + PATH: '' + metadata: + bucket_type: alb + bucket_name: wazuh-alb-integration-tests + +- name: clb_remove_from_bucket + description: CLB remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: clb + BUCKET_NAME: wazuh-clb-integration-tests + PATH: '' + metadata: + bucket_type: clb + bucket_name: wazuh-clb-integration-tests + +- name: nlb_remove_from_bucket + description: NLB remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: nlb + BUCKET_NAME: wazuh-nlb-integration-tests + PATH: '' + metadata: + bucket_type: nlb + bucket_name: wazuh-nlb-integration-tests + +- name: kms_remove_from_bucket + description: KMS remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-kms-integration-tests + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-kms-integration-tests + +- name: macie_remove_from_bucket + description: Macie remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-macie-integration-tests + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-macie-integration-tests + +- name: trusted_advisor_remove_from_bucket + description: Trusted Advisor remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: custom + BUCKET_NAME: wazuh-trusted-advisor-integration-tests + PATH: '' + metadata: + bucket_type: custom + bucket_name: wazuh-trusted-advisor-integration-tests + +- name: guardduty_remove_from_bucket + description: GuardDuty remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-guardduty-integration-tests + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-guardduty-integration-tests + +- name: native_guardduty_remove_from_bucket + description: Native GuardDuty remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: guardduty + BUCKET_NAME: wazuh-native-guardduty-integration-tests + PATH: '' + metadata: + bucket_type: guardduty + bucket_name: wazuh-native-guardduty-integration-tests + +- name: waf_remove_from_bucket + description: WAF remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: waf + BUCKET_NAME: wazuh-waf-integration-tests + PATH: '' + metadata: + bucket_type: waf + bucket_name: wazuh-waf-integration-tests + +- name: server_access_remove_from_bucket + description: Server Access remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: server_access + BUCKET_NAME: wazuh-server-access-integration-tests + PATH: '' + metadata: + bucket_type: server_access + bucket_name: wazuh-server-access-integration-tests + +- name: cisco_umbrella_remove_from_bucket + description: CloudTrail remove from bucket configurations + configuration_parameters: + BUCKET_TYPE: cisco_umbrella + BUCKET_NAME: wazuh-umbrella-integration-tests + PATH: dnslogs + metadata: + bucket_type: cisco_umbrella + bucket_name: wazuh-umbrella-integration-tests + path: dnslogs diff --git a/tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_log_streams.yaml b/tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_log_streams.yaml new file mode 100644 index 0000000000..dfc941327c --- /dev/null +++ b/tests/integration/test_aws/data/test_cases/remove_from_bucket_test_module/cases_remove_log_streams.yaml @@ -0,0 +1,8 @@ +- name: cloudwatchlogs_remove_from_bucket + description: CloudWatch remove from bucket configurations + configuration_parameters: + SERVICE_TYPE: cloudwatchlogs + LOG_GROUP_NAME: temporary-log-group + metadata: + service_type: cloudwatchlogs + log_group_name: temporary-log-group diff --git a/tests/integration/test_aws/test_basic.py b/tests/integration/test_aws/test_basic.py new file mode 100644 index 0000000000..0fd305f997 --- /dev/null +++ b/tests/integration/test_aws/test_basic.py @@ -0,0 +1,315 @@ +import os + +import pytest +from wazuh_testing import TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'basic_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# -------------------------------------------- TEST_BUCKET_DEFAULTS ---------------------------------------------------- +# Configuration and cases data +t1_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'bucket_configuration_defaults.yaml') +t1_cases_path = os.path.join(TEST_CASES_PATH, 'cases_bucket_defaults.yaml') + +# Enabled test configurations +t1_configuration_parameters, t1_configuration_metadata, t1_case_ids = get_test_cases_data(t1_cases_path) +t1_configurations = load_configuration_template( + t1_configurations_path, t1_configuration_parameters, t1_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t1_configurations, t1_configuration_metadata), ids=t1_case_ids) +def test_bucket_defaults( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: The module is invoked with the expected parameters and no error occurs. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check in the ossec.log that no errors occurs. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the log that no errors occurs. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', metadata['bucket_name'], + '--aws_profile', 'qa', + '--type', metadata['bucket_type'], + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + # Detect any ERROR message + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_all_aws_err, + ).result() + + +# -------------------------------------------- TEST_CLOUDWATCH_DEFAULTS ------------------------------------------------ +# Configuration and cases data +t2_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'cloudwatch_configuration_defaults.yaml') +t2_cases_path = os.path.join(TEST_CASES_PATH, 'cases_cloudwatch_defaults.yaml') + +# Enabled test configurations +t2_configuration_parameters, t2_configuration_metadata, t2_case_ids = get_test_cases_data(t2_cases_path) +configurations = load_configuration_template( + t2_configurations_path, t2_configuration_parameters, t2_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, t2_configuration_metadata), ids=t2_case_ids) +def test_service_defaults( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: The module is invoked with the expected parameters and no error occurs. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check in the ossec.log that no errors occurs. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the log that no errors occurs. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + log_groups = metadata.get('log_group_name') + + parameters = [ + 'wodles/aws/aws-s3', + '--service', metadata['service_type'], + '--aws_profile', 'qa', + '--regions', 'us-east-1', + '--aws_log_groups', log_groups, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + # Detect any ERROR message + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_all_aws_err, + ).result() + + +# ------------------------------------------ TEST_INSPECTOR_DEFAULTS --------------------------------------------------- +# Configuration and cases data +t3_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'inspector_configuration_defaults.yaml') +t3_cases_path = os.path.join(TEST_CASES_PATH, 'cases_inspector_defaults.yaml') + +# Enabled test configurations +t3_configuration_parameters, t3_configuration_metadata, t3_case_ids = get_test_cases_data(t3_cases_path) +configurations = load_configuration_template( + t3_configurations_path, t3_configuration_parameters, t3_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, t3_configuration_metadata), ids=t3_case_ids) +def test_inspector_defaults( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: The module is invoked with the expected parameters and no error occurs. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check in the ossec.log that no errors occurs. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the log that no errors occurs. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + + parameters = [ + 'wodles/aws/aws-s3', + '--service', metadata['service_type'], + '--aws_profile', 'qa', + '--regions', 'us-east-1', + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + # Detect any ERROR message + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_all_aws_err, + ).result() diff --git a/tests/integration/test_aws/test_discard_regex.py b/tests/integration/test_aws/test_discard_regex.py new file mode 100644 index 0000000000..0b41bb37c2 --- /dev/null +++ b/tests/integration/test_aws/test_discard_regex.py @@ -0,0 +1,142 @@ +import os + +import pytest +from wazuh_testing import T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.modules.aws.db_utils import s3_db_exists +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'discard_regex_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# ---------------------------------------------------- TEST_PATH ------------------------------------------------------- +configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_discard_regex.yaml') +cases_path = os.path.join(TEST_CASES_PATH, 'cases_discard_regex.yaml') + +configuration_parameters, configuration_metadata, case_ids = get_test_cases_data(cases_path) +configurations = load_configuration_template( + configurations_path, configuration_parameters, configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, configuration_metadata), ids=case_ids) +def test_discard_regex( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring, +): + """ + description: Fetch logs excluding the ones that match with the regex. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check the expected number of events were forwarded to analysisd, only logs stored in the bucket and skips + the ones that match with regex. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly. + input_description: + - The `configuration_discard_regex` file provides the module configuration for this test. + - The `cases_discard_regex` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + bucket_type = metadata['bucket_type'] + only_logs_after = metadata['only_logs_after'] + discard_field = metadata['discard_field'] + discard_regex = metadata['discard_regex'] + found_logs = metadata['found_logs'] + skipped_logs = metadata['skipped_logs'] + path = metadata['path'] if 'path' in metadata else None + + pattern = fr'.*The "{discard_regex}" regex found a match in the "{discard_field}" field. The event will be skipped.' + + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--discard-field', discard_field, + '--discard-regex', discard_regex, + '--type', bucket_type, + '--debug', '2' + ] + + if path is not None: + parameters.insert(5, path) + parameters.insert(5, '--trail_prefix') + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_event_processed_or_skipped(pattern), + error_message=( + 'The AWS module did not show the correct message about discard regex or ', + 'did not process the expected amount of logs' + ), + accum_results=found_logs + skipped_logs + ).result() + + assert s3_db_exists() diff --git a/tests/integration/test_aws/test_log_groups.py b/tests/integration/test_aws/test_log_groups.py new file mode 100644 index 0000000000..fc3a84680a --- /dev/null +++ b/tests/integration/test_aws/test_log_groups.py @@ -0,0 +1,157 @@ +import os + +import pytest +from wazuh_testing import T_10, T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.modules.aws.db_utils import ( + get_multiple_service_db_row, + services_db_exists, + table_exists, +) +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'log_groups_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# ----------------------------------------------- TEST_AWS_LOG_GROUPS -------------------------------------------------- +t1_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_log_groups.yaml') +t1_cases_path = os.path.join(TEST_CASES_PATH, 'cases_log_groups.yaml') + +t1_configuration_parameters, t1_configuration_metadata, t1_case_ids = get_test_cases_data(t1_cases_path) +t1_configurations = load_configuration_template( + t1_configurations_path, t1_configuration_parameters, t1_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t1_configurations, t1_configuration_metadata), ids=t1_case_ids) +def test_log_groups( + configuration, metadata, create_log_stream, load_wazuh_basic_configuration, set_wazuh_configuration, + clean_aws_services_db, configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, + file_monitoring +): + """ + description: Only the events for the specified log_group are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - If a region that does not exist was specified, make sure that a message is displayed in the ossec.log + warning the user. + - Check the expected number of events were forwarded to analysisd, only logs stored in the bucket + for the specified region. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - create_log_stream: + type: fixture + brief: Create a log stream with events for the day of execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly, using the correct path for each entry. + input_description: + - The `configuration_regions` file provides the module configuration for this test. + - The `cases_regions` file provides the test cases. + """ + service_type = metadata['service_type'] + log_group_names = metadata['log_group_name'] + expected_results = metadata['expected_results'] + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--only_logs_after', '2023-JAN-12', + '--regions', 'us-east-1', + '--aws_log_groups', log_group_names, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + if expected_results: + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type), + error_message='The AWS module did not process the expected number of events', + accum_results=len(log_group_names.split(',')) + ).result() + else: + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.make_aws_callback(r'DEBUG: \+\+\+ Sent \d+ events to Analysisd'), + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.make_aws_callback(r'.*The specified log group does not exist.'), + error_message='The AWS module did not show correct message non-existent log group' + ).result() + + assert services_db_exists() + + if expected_results: + log_group_list = log_group_names.split(",") + for row in get_multiple_service_db_row(table_name='cloudwatch_logs'): + assert row.aws_log_group in log_group_list + else: + assert not table_exists(table_name='cloudwatch_logs') diff --git a/tests/integration/test_aws/test_only_logs_after.py b/tests/integration/test_aws/test_only_logs_after.py new file mode 100644 index 0000000000..50d67e894f --- /dev/null +++ b/tests/integration/test_aws/test_only_logs_after.py @@ -0,0 +1,918 @@ +import os +from datetime import datetime + +import pytest +from wazuh_testing import T_10, T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules import aws as cons +from wazuh_testing.modules.aws import ONLY_LOGS_AFTER_PARAM, event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.modules.aws.cli_utils import call_aws_module +from wazuh_testing.modules.aws.cloudwatch_utils import ( + create_log_events, + create_log_stream, +) +from wazuh_testing.modules.aws.db_utils import ( + get_multiple_s3_db_row, + get_s3_db_row, + get_service_db_row, + s3_db_exists, + services_db_exists, +) +from wazuh_testing.modules.aws.s3_utils import get_last_file_key, upload_file +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'only_logs_after_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# --------------------------------------------- TEST_BUCKET_WITHOUT_ONLY_LOGS_AFTER ------------------------------------ +t1_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'bucket_configuration_without_only_logs_after.yaml') +t1_cases_path = os.path.join(TEST_CASES_PATH, 'cases_bucket_without_only_logs_after.yaml') + +t1_configuration_parameters, t1_configuration_metadata, t1_case_ids = get_test_cases_data(t1_cases_path) +t1_configurations = load_configuration_template( + t1_configurations_path, t1_configuration_parameters, t1_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t1_configurations, t1_configuration_metadata), ids=t1_case_ids) +def test_bucket_without_only_logs_after( + configuration, metadata, upload_and_delete_file_to_s3, load_wazuh_basic_configuration, set_wazuh_configuration, + clean_s3_cloudtrail_db, configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, + file_monitoring +): + """ + description: Only the log uploaded during execution is processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check the expected number of events were sent to analysisd. Only the logs whose timestamp is greater than + the date specified in the configuration should be processed. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - upload_and_delete_file_to_s3: + type: fixture + brief: Upload a file for the day of the execution and delete after the test. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the bucket that the uploaded log was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + bucket_type = metadata['bucket_type'] + expected_results = metadata['expected_results'] + table_name = metadata.get('table_name', bucket_type) + path = metadata.get('path') + + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--aws_profile', 'qa', + '--type', bucket_type, + '--debug', '2' + ] + + if path is not None: + parameters.insert(5, path) + parameters.insert(5, '--trail_prefix') + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_event_processed, + error_message='The AWS module did not process the expected number of events', + accum_results=expected_results + ).result() + + assert s3_db_exists() + + data = get_s3_db_row(table_name=table_name) + + assert bucket_name in data.bucket_path + assert metadata['uploaded_file'] == data.log_key + + +# -------------------------------------------- TEST_SERVICE_WITHOUT_ONLY_LOGS_AFTER ------------------------------------ +t2_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'service_configuration_without_only_logs_after.yaml') +t2_cases_path = os.path.join(TEST_CASES_PATH, 'cases_service_without_only_logs_after.yaml') + +t2_configuration_parameters, t2_configuration_metadata, t2_case_ids = get_test_cases_data(t2_cases_path) +t2_configurations = load_configuration_template( + t2_configurations_path, t2_configuration_parameters, t2_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t2_configurations, t2_configuration_metadata), ids=t2_case_ids) +def test_service_without_only_logs_after( + configuration, metadata, create_log_stream_in_existent_group, load_wazuh_basic_configuration, + set_wazuh_configuration, clean_aws_services_db, configure_local_internal_options_function, truncate_monitored_files, + restart_wazuh_function, file_monitoring +): + """ + description: Only the event created during execution is processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check the expected number of events were sent to analysisd. Only the logs whose timestamp is greater than + the date specified in the configuration should be processed. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - create_log_stream_in_existent_group: + type: fixture + brief: Create a log stream with events for the day of execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the bucket that the uploaded log was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + service_type = metadata['service_type'] + log_group_name = metadata['log_group_name'] + expected_results = metadata['expected_results'] + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--regions', 'us-east-1', + '--aws_log_groups', log_group_name, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type), + error_message='The AWS module did not process the expected number of events', + ).result() + + assert services_db_exists() + + data = get_service_db_row(table_name="cloudwatch_logs") + + assert log_group_name == data.aws_log_group + assert metadata['log_stream'] == data.aws_log_stream + + +# --------------------------------------------- TEST_BUCKET_WITH_ONLY_LOGS_AFTER --------------------------------------- +t3_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'bucket_configuration_with_only_logs_after.yaml') +t3_cases_path = os.path.join(TEST_CASES_PATH, 'cases_bucket_with_only_logs_after.yaml') + +t3_configuration_parameters, t3_configuration_metadata, t3_case_ids = get_test_cases_data(t3_cases_path) +t3_configurations = load_configuration_template( + t3_configurations_path, t3_configuration_parameters, t3_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t3_configurations, t3_configuration_metadata), ids=t3_case_ids) +def test_bucket_with_only_logs_after( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: All logs with a timestamp greater than the only_logs_after value are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check the expected number of events were sent to analysisd. Only the logs whose timestamp is greater than + the date specified in the configuration should be processed. + - Check the database was created and updated accordingly + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the bucket that the uploaded log was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + bucket_type = metadata['bucket_type'] + only_logs_after = metadata['only_logs_after'] + expected_results = metadata['expected_results'] + table_name = metadata.get('table_name', bucket_type) + path = metadata.get('path') + + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--type', bucket_type, + '--debug', '2' + ] + + if path is not None: + parameters.insert(5, path) + parameters.insert(5, '--trail_prefix') + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_event_processed, + accum_results=expected_results, + error_message='The AWS module did not process the expected number of events', + ).result() + + assert s3_db_exists() + + for row in get_multiple_s3_db_row(table_name=table_name): + assert bucket_name in row.bucket_path + assert ( + datetime.strptime(only_logs_after, '%Y-%b-%d') < datetime.strptime(str(row.created_date), '%Y%m%d') + ) + + +# --------------------------------------------TEST_CLOUDWATCH_WITH_ONLY_LOGS_AFTER ------------------------------------- +t4_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'cloudwatch_configuration_with_only_logs_after.yaml') +t4_cases_path = os.path.join(TEST_CASES_PATH, 'cases_cloudwatch_with_only_logs_after.yaml') + +t4_configuration_parameters, t4_configuration_metadata, t4_case_ids = get_test_cases_data(t4_cases_path) +t4_configurations = load_configuration_template( + t4_configurations_path, t4_configuration_parameters, t4_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t4_configurations, t4_configuration_metadata), ids=t4_case_ids) +def test_cloudwatch_with_only_logs_after( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: All events with a timestamp greater than the only_logs_after value are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check the expected number of events were sent to analysisd. Only the logs whose timestamp is greater than + the date specified in the configuration should be processed. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the bucket that the uploaded log was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + table_name_map = { + 'inspector': 'aws_services', + 'cloudwatchlogs': 'cloudwatch_logs' + } + + service_type = metadata['service_type'] + log_group_name = metadata.get('log_group_name') + only_logs_after = metadata['only_logs_after'] + expected_results = metadata['expected_results'] + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--regions', 'us-east-1', + '--aws_log_groups', log_group_name, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type), + error_message='The AWS module did not process the expected number of events', + ).result() + + assert services_db_exists() + + data = get_service_db_row(table_name=table_name_map[service_type]) + + assert log_group_name == data.aws_log_group + assert metadata['log_stream'] == data.aws_log_stream + + +# ------------------------------------------ TEST_INSPECTOR_WITH_ONLY_LOGS_AFTER --------------------------------------- +t5_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'inspector_configuration_with_only_logs_after.yaml') +t5_cases_path = os.path.join(TEST_CASES_PATH, 'cases_inspector_with_only_logs_after.yaml') + +t5_configuration_parameters, t5_configuration_metadata, t5_case_ids = get_test_cases_data(t5_cases_path) +t5_configurations = load_configuration_template( + t5_configurations_path, t5_configuration_parameters, t5_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t5_configurations, t5_configuration_metadata), ids=t5_case_ids) +def test_inspector_with_only_logs_after( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: All events with a timestamp greater than the only_logs_after value are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check the expected number of events were sent to analysisd. Only the logs whose timestamp is greater than + the date specified in the configuration should be processed. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the bucket that the uploaded log was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + table_name_map = { + 'inspector': 'aws_services', + 'cloudwatchlogs': 'cloudwatch_logs' + } + + service_type = metadata['service_type'] + only_logs_after = metadata['only_logs_after'] + expected_results = metadata['expected_results'] + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--regions', 'us-east-1', + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type), + error_message='The AWS module did not process the expected number of events', + ).result() + + assert services_db_exists() + + data = get_service_db_row(table_name=table_name_map[service_type]) + + assert data.service == service_type + assert ( + datetime.strptime(data.timestamp, '%Y-%m-%d %H:%M:%S.%f') == datetime.strptime(only_logs_after, '%Y-%b-%d') + ) + + +# ---------------------------------------------------- TEST_MULTIPLE_CALLS --------------------------------------------- +t5_cases_path = os.path.join(TEST_CASES_PATH, 'cases_bucket_multiple_calls.yaml') + +_, t5_configuration_metadata, t5_case_ids = get_test_cases_data(t5_cases_path) + + +@pytest.mark.tier(level=1) +@pytest.mark.parametrize('metadata', t5_configuration_metadata, ids=t5_case_ids) +def test_bucket_multiple_calls( + metadata, clean_s3_cloudtrail_db, load_wazuh_basic_configuration, restart_wazuh_function, delete_file_from_s3 +): + """ + description: Call the AWS module multiple times with different only_logs_after values. + test_phases: + - setup: + - Delete the `s3_cloudtrail.db`. + + - test: + - Call the module without only_logs_after and check that no logs were processed. + - Upload a log file for the day of the test execution and call the module with the same parameters as + before, check that the uploaded logs were processed. + - Call the module with the same parameters and check that no logs were processed, there were no duplicates. + - Call the module with only_logs_after set in the past and check that the expected number of logs were + processed. + - Call the module with the same parameters in and check there were no duplicates. + - Call the module with only_logs_after set with an older date check that old logs were processed without + duplicates. + - Call the module with only_logs_after set with an early date than setted previously and check that no logs + were processed, there were no duplicates. + + - teardown: + - Delete the `s3_cloudtrail.db`. + - Delete the uploaded files. + wazuh_min_version: 4.5.0 + parameters: + - metadata: + type: dict + brief: Get metadata from the module. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - restart_wazuh_daemon: + type: fixture + brief: Restart the wazuh service. + - delete_file_from_s3: + type: fixture + brief: Delete the a file after the test execution. + input_description: + - The `cases_multiple_calls` file provides the test cases. + """ + + bucket_type = metadata['bucket_type'] + bucket_name = metadata['bucket_name'] + path = metadata.get('path') + + base_parameters = [ + '--bucket', bucket_name, + '--type', bucket_type, + '--regions', 'us-east-1', + '--aws_profile', 'qa', + '--debug', '2' + ] + + if path is not None: + base_parameters.extend(['--trail_prefix', path]) + + # Call the module without only_logs_after and check that no logs were processed + last_marker_key = datetime.utcnow().strftime(cons.PATH_DATE_FORMAT) + + event_monitor.check_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters), + bucket_type=bucket_type + ) + + # Call the module with only_logs_after set in the past and check that the expected number of logs were + # processed + event_monitor.check_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2022-NOV-20'), + expected_results=3 + ) + + # Call the module with the same parameters in and check there were no duplicates + expected_skipped_logs_step_3 = metadata.get('expected_skipped_logs_step_3', 1) + event_monitor.check_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2022-NOV-20'), + bucket_type=bucket_type, + expected_results=expected_skipped_logs_step_3 + ) + + # Call the module with only_logs_after set with an early date than setted previously and check that no logs + # were processed, there were no duplicates + event_monitor.check_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2022-NOV-22'), + bucket_type=bucket_type, + expected_results=expected_skipped_logs_step_3 - 1 if expected_skipped_logs_step_3 > 1 else 1 + ) + + # Upload a log file for the day of the test execution and call the module without only_logs_after and check that + # only the uploaded logs were processed and the last marker is specified in the DB. + last_marker_key = get_last_file_key(bucket_type, bucket_name, datetime.utcnow()) + metadata['filename'] = upload_file(bucket_type, bucket_name) + + event_monitor.check_marker_from_output( + command_output=call_aws_module(*base_parameters), + file_key=last_marker_key + ) + + +# -------------------------------------------- TEST_INSPECTOR_MULTIPLE_CALLS ------------------------------------------- +t6_cases_path = os.path.join(TEST_CASES_PATH, 'cases_inspector_multiple_calls.yaml') + +_, t6_configuration_metadata, t6_case_ids = get_test_cases_data(t6_cases_path) + + +@pytest.mark.tier(level=1) +@pytest.mark.parametrize('metadata', t6_configuration_metadata, ids=t6_case_ids) +@pytest.mark.xfail +def test_inspector_multiple_calls( + metadata, clean_aws_services_db, load_wazuh_basic_configuration, restart_wazuh_function +): + """ + description: Call the AWS module multiple times with different only_logs_after values. + test_phases: + - setup: + - Delete the `aws_services.db`. + - test: + - Call the module without only_logs_after and check that no logs were processed. + - Call the module with only_logs_after set in the past and check that the expected number of logs were + processed. + - Call the module with the same parameters in and check there were no duplicates. + - Call the module with only_logs_after set with an early date than setted previously and check that no logs + were processed, there were no duplicates. + - teardown: + - Delete the `aws_services.db`. + wazuh_min_version: 4.5.0 + parameters: + - metadata: + type: dict + brief: Get metadata from the module. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - restart_wazuh_daemon: + type: fixture + brief: Restart the wazuh service. + input_description: + - The `cases_multiple_calls` file provides the test cases. + """ + + service_type = metadata['service_type'] + + base_parameters = [ + '--service', service_type, + '--regions', 'us-east-1', + '--aws_profile', 'qa', + '--debug', '2' + ] + + # Call the module without only_logs_after and check that no logs were processed + event_monitor.check_service_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters), service_type=service_type, expected_results=1 + ) + + # Call the module with only_logs_after set in the past and check that the expected number of logs were + # processed + event_monitor.check_service_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2023-JAN-30'), + service_type=service_type, + events_sent=4 + ) + + # Call the module with the same parameters in and check there were no duplicates + event_monitor.check_service_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2023-JAN-30'), + service_type=service_type, + expected_results=1 + ) + + # Call the module with only_logs_after set with an early date than setted previously and check that no logs + # were processed, there were no duplicates + event_monitor.check_service_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2023-JAN-31'), + service_type=service_type, + expected_results=1 + ) + + +# ----------------------------------------- TEST_CLOUDWATCH_MULTIPLE_CALLS --------------------------------------------- +t7_cases_path = os.path.join(TEST_CASES_PATH, 'cases_cloudwatch_multiple_calls.yaml') + +_, t7_configuration_metadata, t7_case_ids = get_test_cases_data(t7_cases_path) + + +@pytest.mark.tier(level=1) +@pytest.mark.parametrize('metadata', t7_configuration_metadata, ids=t7_case_ids) +def test_cloudwatch_multiple_calls( + metadata, clean_aws_services_db, load_wazuh_basic_configuration, restart_wazuh_function, delete_log_stream +): + """ + description: Call the AWS module multiple times with different only_logs_after values. + test_phases: + - setup: + - Delete the `aws_services.db`. + - test: + - Call the module without only_logs_after and check that no logs were processed. + - Upload a log file for the day of the test execution and call the module with the same parameters as + before, check that the uploaded logs were processed. + - Call the module with the same parameters and check that no logs were processed, there were no duplicates. + - Call the module with only_logs_after set in the past and check that the expected number of logs were + processed. + - Call the module with the same parameters in and check there were no duplicates. + - Call the module with only_logs_after set with an older date check that old logs were processed without + duplicates. + - Call the module with only_logs_after set with an early date than setted previously and check that no logs + were processed, there were no duplicates. + - teardown: + - Delete the `aws_services.db`. + - Delete the uploaded files. + wazuh_min_version: 4.5.0 + parameters: + - metadata: + type: dict + brief: Get metadata from the module. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - restart_wazuh_daemon: + type: fixture + brief: Restart the wazuh service. + - delete_log_stream: + type: fixture + brief: Delete the log stream after the test execution. + input_description: + - The `cases_multiple_calls` file provides the test cases. + """ + + service_type = metadata['service_type'] + log_group_name = metadata['log_group_name'] + + base_parameters = [ + '--service', service_type, + '--aws_log_groups', log_group_name, + '--regions', 'us-east-1', + '--aws_profile', 'qa', + '--debug', '2' + ] + + # Call the module without only_logs_after and check that no logs were processed + event_monitor.check_service_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters), service_type=service_type, expected_results=0 + ) + + # Call the module with only_logs_after set in the past and check that the expected number of logs were + # processed + event_monitor.check_service_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2023-JAN-12'), + service_type=service_type, + events_sent=3 + ) + + # Call the module with the same parameters in and check there were no duplicates + event_monitor.check_service_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2023-JAN-12'), + service_type=service_type, + expected_results=0 + ) + + # Call the module with only_logs_after set with an early date than setted previously and check that no logs + # were processed, there were no duplicates + event_monitor.check_service_non_processed_logs_from_output( + command_output=call_aws_module(*base_parameters, ONLY_LOGS_AFTER_PARAM, '2023-JAN-15'), + service_type=service_type, + expected_results=0 + ) + + # Upload a log file for the day of the test execution and call the module without only_logs_after and check that + # only the uploaded logs were processed. + log_stream = create_log_stream() + metadata['log_stream'] = log_stream + create_log_events(log_stream) + event_monitor.check_service_processed_logs_from_output( + command_output=call_aws_module(*base_parameters), service_type=service_type, events_sent=1 + ) diff --git a/tests/integration/test_aws/test_parser.py b/tests/integration/test_aws/test_parser.py new file mode 100644 index 0000000000..25011f87c9 --- /dev/null +++ b/tests/integration/test_aws/test_parser.py @@ -0,0 +1,597 @@ +import os + +import pytest +from wazuh_testing import T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'parser_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# --------------------------------------------TEST_BUCKET_AND_SERVICE_MISSING ------------------------------------------ +# Configuration and cases data +t1_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_bucket_and_service_missing.yaml') +t1_cases_path = os.path.join(TEST_CASES_PATH, 'cases_bucket_and_service_missing.yaml') + +# Enabled test configurations +t1_configuration_parameters, t1_configuration_metadata, t1_case_ids = get_test_cases_data(t1_cases_path) +t1_configurations = load_configuration_template( + t1_configurations_path, t1_configuration_parameters, t1_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t1_configurations, t1_configuration_metadata), ids=t1_case_ids) +def test_bucket_and_service_missing( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: Command for bucket and service weren't invoked. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was not called. + + input_description: + - The `configuration_bucket_and_service_missing` file provides the configuration for this test. + """ + + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_warning, + error_message='The AWS module did not show the expected warning', + ).result() + + # Check AWS module not started + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_started, + ).result() + + +# -------------------------------------------- TEST_TYPE_MISSING_IN_BUCKET --------------------------------------------- +# Configuration and cases data +t2_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_type_missing_in_bucket.yaml') +t2_cases_path = os.path.join(TEST_CASES_PATH, 'cases_type_missing_in_bucket.yaml') + +# Enabled test configurations +t2_configuration_parameters, t2_configuration_metadata, t2_case_ids = get_test_cases_data(t2_cases_path) +t2_configurations = load_configuration_template( + t2_configurations_path, t2_configuration_parameters, t2_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t2_configurations, t2_configuration_metadata), ids=t2_case_ids) +def test_type_missing_in_bucket( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: A warning occurs and was displayed in `ossec.log`. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module displays the message about missing attributes. + input_description: + - The `configuration_type_missing_in_bucket` file provides the configuration for this test. + """ + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_legacy_module_warning, + error_message='The AWS module did not show the expected legacy warning', + ).result() + + +# -------------------------------------------- TEST_TYPE_MISSING_IN_SERVICE -------------------------------------------- +# Configuration and cases data +t3_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_type_missing_in_service.yaml') +t3_cases_path = os.path.join(TEST_CASES_PATH, 'cases_type_missing_in_service.yaml') + +# Enabled test configurations +t3_configuration_parameters, t3_configuration_metadata, t3_case_ids = get_test_cases_data(t3_cases_path) +t3_configurations = load_configuration_template( + t3_configurations_path, t3_configuration_parameters, t3_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t3_configurations, t3_configuration_metadata), ids=t3_case_ids) +def test_type_missing_in_service( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: An error occurs and was displayed in `ossec.log`. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module displays the message about missing attributes. + + input_description: + - The `configuration_type_missing_in_service` file provides the configuration for this test. + """ + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_error_for_missing_type, + error_message='The AWS module did not show the expected error message', + ).result() + + +# -------------------------------------------- TEST_EMPTY_VALUES_IN_BUCKET --------------------------------------------- +# Configuration and cases data +t4_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_values_in_bucket.yaml') +t4_cases_path = os.path.join(TEST_CASES_PATH, 'cases_empty_values_in_bucket.yaml') + +# Enabled test configurations +t4_configuration_parameters, t4_configuration_metadata, t4_case_ids = get_test_cases_data(t4_cases_path) +t4_configurations = load_configuration_template( + t4_configurations_path, t4_configuration_parameters, t4_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t4_configurations, t4_configuration_metadata), ids=t4_case_ids) +def test_empty_values_in_bucket( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: An error occurs and was displayed in `ossec.log`. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module displays the message about an empty value. + input_description: + - The `configuration_values_in_bucket` file provides the configuration for this test. + """ + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_empty_value, + error_message='The AWS module did not show the expected message about empty value', + ).result() + + +# -------------------------------------------- TEST_EMPTY_VALUES_IN_SERVICE -------------------------------------------- +# Configuration and cases data +t5_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_values_in_service.yaml') +t5_cases_path = os.path.join(TEST_CASES_PATH, 'cases_empty_values_in_service.yaml') + +# Enabled test configurations +t5_configuration_parameters, t5_configuration_metadata, t5_case_ids = get_test_cases_data(t5_cases_path) +t5_configurations = load_configuration_template( + t5_configurations_path, t5_configuration_parameters, t5_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t5_configurations, t5_configuration_metadata), ids=t5_case_ids) +def test_empty_values_in_service( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: An error occurs and was displayed in `ossec.log`. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module displays the message about an empty value. + + input_description: + - The `configuration_values_in_service` file provides the configuration for this test. + """ + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_empty_value, + error_message='The AWS module did not show the expected message about empty value', + ).result() + + +# ------------------------------------------ TEST_INVALID_VALUES_IN_BUCKET --------------------------------------------- +# Configuration and cases data +t6_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_values_in_bucket.yaml') +t6_cases_path = os.path.join(TEST_CASES_PATH, 'cases_invalid_values_in_bucket.yaml') + +# Enabled test configurations +t6_configuration_parameters, t6_configuration_metadata, t6_case_ids = get_test_cases_data(t6_cases_path) +t6_configurations = load_configuration_template( + t6_configurations_path, t6_configuration_parameters, t6_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t6_configurations, t6_configuration_metadata), ids=t6_case_ids) +def test_invalid_values_in_bucket( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: An error occurs and was displayed in `ossec.log`. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module displays the message about an invalid value. + input_description: + - The `configuration_values_in_bucket` file provides the configuration for this test. + """ + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_invalid_value, + error_message='The AWS module did not show the expected message about invalid value', + ).result() + + +# ------------------------------------------ TEST_INVALID_VALUES_IN_BUCKET --------------------------------------------- +# Configuration and cases data +t7_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_values_in_service.yaml') +t7_cases_path = os.path.join(TEST_CASES_PATH, 'cases_invalid_values_in_service.yaml') + +# Enabled test configurations +t7_configuration_parameters, t7_configuration_metadata, t7_case_ids = get_test_cases_data(t7_cases_path) +t7_configurations = load_configuration_template( + t7_configurations_path, t7_configuration_parameters, t7_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t7_configurations, t7_configuration_metadata), ids=t7_case_ids) +def test_invalid_values_in_service( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: An error occurs and was displayed in `ossec.log`. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module displays the message about an invalid value. + input_description: + - The `configuration_values_in_service` file provides the configuration for this test. + """ + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_invalid_value, + error_message='The AWS module did not show the expected message about invalid value', + ).result() + + +# --------------------------------------- TEST_MULTIPLE_BUCKET_AND_SERVICE_TAGS ---------------------------------------- +# Configuration and cases data +t8_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_multiple_bucket_and_service_tags.yaml') +t8_cases_path = os.path.join(TEST_CASES_PATH, 'cases_multiple_bucket_and_service_tags.yaml') + +# Enabled test configurations +t8_configuration_parameters, t8_configuration_metadata, t8_case_ids = get_test_cases_data(t8_cases_path) +t8_configurations = load_configuration_template( + t8_configurations_path, t8_configuration_parameters, t8_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t8_configurations, t8_configuration_metadata), ids=t8_case_ids) +def test_multiple_bucket_and_service_tags( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function_without_exception, + file_monitoring +): + """ + description: The command is invoked two times for buckets and two times for services. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has not appeared calling the module with correct parameters. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_function_without_exception: + type: fixture + brief: Restart the wazuh service catching the exception. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called the right amount of times. + input_description: + - The `configuration_multiple_bucket_and_service_tags` file provides the configuration for this test. + """ + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_bucket_or_service_call, + error_message='The AWS module was not called for bucket or service the right amount of times', + accum_results=4 + ).result() diff --git a/tests/integration/test_aws/test_path.py b/tests/integration/test_aws/test_path.py new file mode 100644 index 0000000000..6d377217ed --- /dev/null +++ b/tests/integration/test_aws/test_path.py @@ -0,0 +1,155 @@ +import os + +import pytest +from wazuh_testing import T_10, T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.modules.aws.db_utils import ( + get_s3_db_row, + s3_db_exists, + table_exists_or_has_values, +) +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'path_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# ---------------------------------------------------- TEST_PATH ------------------------------------------------------- +configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_path.yaml') +cases_path = os.path.join(TEST_CASES_PATH, 'cases_path.yaml') + +configuration_parameters, configuration_metadata, case_ids = get_test_cases_data(cases_path) +configurations = load_configuration_template( + configurations_path, configuration_parameters, configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, configuration_metadata), ids=case_ids) +def test_path( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: Only logs within a path are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - If a path that does not exist was specified, make sure that a message is displayed in the ossec.log + warning the user. + - Check the command was called with the correct parameters. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly, using the correct path for each entry. + input_description: + - The `configuration_path` file provides the module configuration for this test. + - The `cases_path` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + bucket_type = metadata['bucket_type'] + only_logs_after = metadata['only_logs_after'] + path = metadata['path'] + expected_results = metadata['expected_results'] + table_name = metadata.get('table_name', bucket_type) + pattern = fr".*WARNING: Bucket: - No files were found in '{bucket_name}/{path}/'. No logs will be processed.\n+" + + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--aws_profile', 'qa', + '--trail_prefix', path, + '--only_logs_after', only_logs_after, + '--type', bucket_type, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + if expected_results: + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_event_processed, + error_message='The AWS module did not process the expected number of events', + ).result() + else: + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_event_processed, + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.make_aws_callback(pattern), + error_message='The AWS module did not show correct message about empty path' + ).result() + + assert s3_db_exists() + + if expected_results: + data = get_s3_db_row(table_name=table_name) + assert f"{bucket_name}/{path}/" == data.bucket_path + assert data.log_key.startswith(f"{path}/") + else: + assert not table_exists_or_has_values(table_name=table_name) diff --git a/tests/integration/test_aws/test_path_suffix.py b/tests/integration/test_aws/test_path_suffix.py new file mode 100644 index 0000000000..f9597edb34 --- /dev/null +++ b/tests/integration/test_aws/test_path_suffix.py @@ -0,0 +1,157 @@ +import os + +import pytest +from wazuh_testing import T_10, T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.modules.aws.db_utils import ( + get_s3_db_row, + s3_db_exists, + table_exists_or_has_values, +) +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'path_suffix_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# ---------------------------------------------------- TEST_PATH ------------------------------------------------------- +configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_path_suffix.yaml') +cases_path = os.path.join(TEST_CASES_PATH, 'cases_path_suffix.yaml') + +configuration_parameters, configuration_metadata, case_ids = get_test_cases_data(cases_path) +configurations = load_configuration_template( + configurations_path, configuration_parameters, configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, configuration_metadata), ids=case_ids) +def test_path_suffix( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: Only logs within a path_suffix are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - If a path_suffix that does not exist was specified, make sure that a message is displayed in the ossec.log + warning the user. + - Check the command was called with the correct parameters. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly, using the correct path for each entry. + input_description: + - The `configuration_path_suffix` file provides the module configuration for this test. + - The `cases_path_suffix` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + bucket_type = metadata['bucket_type'] + only_logs_after = metadata['only_logs_after'] + path_suffix = metadata['path_suffix'] + expected_results = metadata['expected_results'] + pattern = ( + fr".*No logs found in 'AWSLogs/{path_suffix}/'. " + fr"Check the provided prefix and the location of the logs for the bucket type '{bucket_type}'*" + ) + + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--aws_profile', 'qa', + '--trail_suffix', path_suffix, + '--only_logs_after', only_logs_after, + '--type', bucket_type, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + if expected_results: + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_event_processed, + error_message='The AWS module did not process the expected number of events', + ).result() + else: + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_event_processed, + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.make_aws_callback(pattern), + error_message='The AWS module did not show correct message about empty path_suffix' + ).result() + + assert s3_db_exists() + + if expected_results: + data = get_s3_db_row(table_name=bucket_type) + assert f"{bucket_name}/{path_suffix}/" == data.bucket_path + assert data.log_key.startswith(f"AWSLogs/{path_suffix}/") + else: + assert not table_exists_or_has_values(table_name=bucket_type) diff --git a/tests/integration/test_aws/test_regions.py b/tests/integration/test_aws/test_regions.py new file mode 100644 index 0000000000..920dca3253 --- /dev/null +++ b/tests/integration/test_aws/test_regions.py @@ -0,0 +1,443 @@ +import os + +import pytest +from wazuh_testing import T_10, T_20, TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import ( # noqa: F401 + AWS_SERVICES_DB_PATH, + RANDOM_ACCOUNT_ID, + event_monitor, + local_internal_options +) +from wazuh_testing.modules.aws.db_utils import ( + get_multiple_s3_db_row, + get_multiple_service_db_row, + s3_db_exists, + table_exists_or_has_values, +) +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'regions_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# ---------------------------------------------------- TEST_PATH ------------------------------------------------------- +t1_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'bucket_configuration_regions.yaml') +t1_cases_path = os.path.join(TEST_CASES_PATH, 'cases_bucket_regions.yaml') + +t1_configuration_parameters, t1_configuration_metadata, t1_case_ids = get_test_cases_data(t1_cases_path) +t1_configurations = load_configuration_template( + t1_configurations_path, t1_configuration_parameters, t1_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t1_configurations, t1_configuration_metadata), ids=t1_case_ids) +def test_regions( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_s3_cloudtrail_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: Only the logs for the specified region are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - If a region that does not exist was specified, make sure that a message is displayed in the ossec.log + warning the user. + - Check the expected number of events were forwarded to analysisd, only logs stored in the bucket + for the specified region. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly, using the correct path for each entry. + input_description: + - The `configuration_regions` file provides the module configuration for this test. + - The `cases_regions` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + bucket_type = metadata['bucket_type'] + only_logs_after = metadata['only_logs_after'] + regions = metadata['regions'] + expected_results = metadata['expected_results'] + pattern = fr".*DEBUG: \+\+\+ No logs to process in bucket: {RANDOM_ACCOUNT_ID}/{regions}" + + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--regions', regions, + '--type', bucket_type, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + if expected_results: + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_event_processed, + error_message='The AWS module did not process the expected number of events', + accum_results=expected_results + ).result() + else: + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_event_processed, + ).result() + + log_monitor.start( + timeout=T_10, + callback=event_monitor.make_aws_callback(pattern), + error_message='The AWS module did not show correct message about non-existent region' + ).result() + + assert s3_db_exists() + + if expected_results: + regions_list = regions.split(",") + for row in get_multiple_s3_db_row(table_name=bucket_type): + if hasattr(row, "aws_region"): + assert row.aws_region in regions_list + else: + assert row.log_key.split("/")[3] in regions_list + else: + assert not table_exists_or_has_values(table_name=bucket_type) + + +# -------------------------------------------- TEST_CLOUDWATCH_REGIONS ------------------------------------------------- +t2_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'cloudwatch_configuration_regions.yaml') +t2_cases_path = os.path.join(TEST_CASES_PATH, 'cases_cloudwatch_regions.yaml') + +t2_configuration_parameters, t2_configuration_metadata, t2_case_ids = get_test_cases_data(t2_cases_path) +configurations = load_configuration_template( + t2_configurations_path, t2_configuration_parameters, t2_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, t2_configuration_metadata), ids=t2_case_ids) +def test_cloudwatch_regions( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: Only the logs for the specified region are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - If a region that does not exist was specified, make sure that a message is displayed in the ossec.log + warning the user. + - Check the expected number of events were forwarded to analysisd, only logs stored in the bucket + for the specified region. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly, using the correct path for each entry. + input_description: + - The `configuration_regions` file provides the module configuration for this test. + - The `cases_regions` file provides the test cases. + """ + service_type = metadata['service_type'] + log_group_name = metadata.get('log_group_name') + only_logs_after = metadata['only_logs_after'] + regions: str = metadata['regions'] + expected_results = metadata['expected_results'] + regions_list = regions.split(",") + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--regions', regions, + '--aws_log_groups', log_group_name, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + if expected_results: + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type), + error_message='The AWS module did not process the expected number of events', + accum_results=len(regions_list) + ).result() + else: + pattern = ( + r'DEBUG: \+\+\+ \d+ events collected and processed in' if service_type == 'inspector' + else r'DEBUG: \+\+\+ Sent \d+ events to Analysisd' + ) + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.make_aws_callback(pattern), + ).result() + + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.make_aws_callback( + fr".*\+\+\+ ERROR: The region '{regions}' is not a valid one." + ), + error_message='The AWS module did not show correct message non-existent region' + ).result() + + table_name = 'cloudwatch_logs' + + if expected_results: + assert table_exists_or_has_values(table_name=table_name, db_path=AWS_SERVICES_DB_PATH) + for row in get_multiple_service_db_row(table_name=table_name): + assert (getattr(row, 'region', None) or getattr(row, 'aws_region')) in regions_list + else: + assert not table_exists_or_has_values(table_name=table_name, db_path=AWS_SERVICES_DB_PATH) + + +# ------------------------------------------ TEST_INSPECTOR_PATH ------------------------------------------------------- +t3_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'inspector_configuration_regions.yaml') +t3_cases_path = os.path.join(TEST_CASES_PATH, 'cases_inspector_regions.yaml') + +t3_configuration_parameters, t3_configuration_metadata, t3_case_ids = get_test_cases_data(t3_cases_path) +configurations = load_configuration_template( + t3_configurations_path, t3_configuration_parameters, t3_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(configurations, t3_configuration_metadata), ids=t3_case_ids) +def test_inspector_regions( + configuration, metadata, load_wazuh_basic_configuration, set_wazuh_configuration, clean_aws_services_db, + configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: Only the logs for the specified region are processed. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - If a region that does not exist was specified, make sure that a message is displayed in the ossec.log + warning the user. + - Check the expected number of events were forwarded to analysisd, only logs stored in the bucket + for the specified region. + - Check the database was created and updated accordingly. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + - Delete the uploaded file. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check the expected number of events were forwarded to analysisd. + - Check the database was created and updated accordingly, using the correct path for each entry. + input_description: + - The `configuration_regions` file provides the module configuration for this test. + - The `cases_regions` file provides the test cases. + """ + service_type = metadata['service_type'] + only_logs_after = metadata['only_logs_after'] + regions: str = metadata['regions'] + expected_results = metadata['expected_results'] + regions_list = regions.split(",") + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--only_logs_after', only_logs_after, + '--regions', regions, + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + if expected_results: + log_monitor.start( + timeout=T_20, + callback=event_monitor.callback_detect_service_event_processed(expected_results, service_type), + error_message='The AWS module did not process the expected number of events', + accum_results=len(regions_list) + ).result() + else: + pattern = ( + r'DEBUG: \+\+\+ \d+ events collected and processed in' if service_type == 'inspector' + else r'DEBUG: \+\+\+ Sent \d+ events to Analysisd' + ) + with pytest.raises(TimeoutError): + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.make_aws_callback(pattern), + ).result() + + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.make_aws_callback( + fr".*\+\+\+ ERROR: The region '{regions}' is not a valid one." + ), + error_message='The AWS module did not show correct message non-existent region' + ).result() + + table_name = 'aws_services' + + if expected_results: + assert table_exists_or_has_values(table_name=table_name, db_path=AWS_SERVICES_DB_PATH) + for row in get_multiple_service_db_row(table_name=table_name): + assert (getattr(row, 'region', None) or getattr(row, 'aws_region')) in regions_list + else: + assert not table_exists_or_has_values(table_name=table_name, db_path=AWS_SERVICES_DB_PATH) diff --git a/tests/integration/test_aws/test_remove_from_bucket.py b/tests/integration/test_aws/test_remove_from_bucket.py new file mode 100644 index 0000000000..8bddca8fb1 --- /dev/null +++ b/tests/integration/test_aws/test_remove_from_bucket.py @@ -0,0 +1,225 @@ +import os + +import pytest +from wazuh_testing import TEMPLATE_DIR, TEST_CASES_DIR, global_parameters +from wazuh_testing.modules.aws import event_monitor, local_internal_options # noqa: F401 +from wazuh_testing.modules.aws.cloudwatch_utils import log_stream_exists +from wazuh_testing.modules.aws.s3_utils import file_exists +from wazuh_testing.tools.configuration import ( + get_test_cases_data, + load_configuration_template, +) + +pytestmark = [pytest.mark.server] + + +# Generic vars +MODULE = 'remove_from_bucket_test_module' +TEST_DATA_PATH = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'data') +CONFIGURATIONS_PATH = os.path.join(TEST_DATA_PATH, TEMPLATE_DIR, MODULE) +TEST_CASES_PATH = os.path.join(TEST_DATA_PATH, TEST_CASES_DIR, MODULE) + +# ---------------------------------------------------- TEST_REMOVE_FROM_BUCKET ----------------------------------------- +# Configuration and cases data +t1_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_remove_from_bucket.yaml') +t1_cases_path = os.path.join(TEST_CASES_PATH, 'cases_remove_from_bucket.yaml') + +t1_configuration_parameters, t1_configuration_metadata, t1_case_ids = get_test_cases_data(t1_cases_path) +t1_configurations = load_configuration_template( + t1_configurations_path, t1_configuration_parameters, t1_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t1_configurations, t1_configuration_metadata), ids=t1_case_ids) +def test_remove_from_bucket( + configuration, metadata, mark_cases_as_skipped, upload_and_delete_file_to_s3, load_wazuh_basic_configuration, + set_wazuh_configuration, clean_s3_cloudtrail_db, configure_local_internal_options_function, + truncate_monitored_files, restart_wazuh_function, file_monitoring +): + """ + description: The uploaded file was removed after the execution. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check that the uploaded log was removed by the module after the execution. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - upload_and_delete_file_to_s3: + type: fixture + brief: Upload a file to S3 bucket for the day of the execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_s3_cloudtrail_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the bucket that the uploaded log was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + bucket_name = metadata['bucket_name'] + path = metadata.get('path') + parameters = [ + 'wodles/aws/aws-s3', + '--bucket', bucket_name, + '--remove', + '--aws_profile', 'qa', + '--type', metadata['bucket_type'], + '--debug', '2' + ] + + if path is not None: + parameters.insert(6, path) + parameters.insert(6, '--trail_prefix') + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + assert not file_exists(filename=metadata['uploaded_file'], bucket_name=bucket_name) + + +# ---------------------------------------------------- TEST_REMOVE_LOG_STREAM ------------------------------------------ +# Configuration and cases data +t2_configurations_path = os.path.join(CONFIGURATIONS_PATH, 'configuration_remove_log_stream.yaml') +t2_cases_path = os.path.join(TEST_CASES_PATH, 'cases_remove_log_streams.yaml') + +t2_configuration_parameters, t2_configuration_metadata, t2_case_ids = get_test_cases_data(t2_cases_path) +t2_configurations = load_configuration_template( + t2_configurations_path, t2_configuration_parameters, t2_configuration_metadata +) + + +@pytest.mark.tier(level=0) +@pytest.mark.parametrize('configuration, metadata', zip(t2_configurations, t2_configuration_metadata), ids=t2_case_ids) +def test_remove_log_stream( + configuration, metadata, create_log_stream, load_wazuh_basic_configuration, set_wazuh_configuration, + clean_aws_services_db, configure_local_internal_options_function, truncate_monitored_files, restart_wazuh_function, + file_monitoring +): + """ + description: The created log stream was removed after the execution. + test_phases: + - setup: + - Load Wazuh light configuration. + - Apply ossec.conf configuration changes according to the configuration template and use case. + - Apply custom settings in local_internal_options.conf. + - Truncate wazuh logs. + - Restart wazuh-manager service to apply configuration changes. + - test: + - Check in the ossec.log that a line has appeared calling the module with correct parameters. + - Check that the created log stream was removed by the module after the execution. + - teardown: + - Truncate wazuh logs. + - Restore initial configuration, both ossec.conf and local_internal_options.conf. + wazuh_min_version: 4.5.0 + parameters: + - configuration: + type: dict + brief: Get configurations from the module. + - metadata: + type: dict + brief: Get metadata from the module. + - create_log_stream: + type: fixture + brief: Create a log stream with events for the day of execution. + - load_wazuh_basic_configuration: + type: fixture + brief: Load basic wazuh configuration. + - set_wazuh_configuration: + type: fixture + brief: Apply changes to the ossec.conf configuration. + - clean_aws_services_db: + type: fixture + brief: Delete the DB file before and after the test execution. + - configure_local_internal_options_function: + type: fixture + brief: Apply changes to the local_internal_options.conf configuration. + - truncate_monitored_files: + type: fixture + brief: Truncate wazuh logs. + - restart_wazuh_daemon_function: + type: fixture + brief: Restart the wazuh service. + - file_monitoring: + type: fixture + brief: Handle the monitoring of a specified file. + assertions: + - Check in the log that the module was called with correct parameters. + - Check in the log group that the created stream was removed. + input_description: + - The `configuration_defaults` file provides the module configuration for this test. + - The `cases_defaults` file provides the test cases. + """ + service_type = metadata['service_type'] + log_group_name = metadata['log_group_name'] + + parameters = [ + 'wodles/aws/aws-s3', + '--service', service_type, + '--aws_profile', 'qa', + '--regions', 'us-east-1', + '--aws_log_groups', log_group_name, + '--remove-log-streams', + '--debug', '2' + ] + + # Check AWS module started + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_start, + error_message='The AWS module did not start as expected', + ).result() + + # Check command was called correctly + log_monitor.start( + timeout=global_parameters.default_timeout, + callback=event_monitor.callback_detect_aws_module_called(parameters), + error_message='The AWS module was not called with the correct parameters', + ).result() + + assert not log_stream_exists(log_stream=metadata['log_stream'], log_group=log_group_name)