Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update AWS Services to make use of the discard_regex functionality #17388

Closed
7 tasks done
fdalmaup opened this issue Jun 2, 2023 · 7 comments · Fixed by #17748, wazuh/wazuh-qa#4278 or wazuh/wazuh-documentation#6207
Closed
7 tasks done

Update AWS Services to make use of the discard_regex functionality #17388

fdalmaup opened this issue Jun 2, 2023 · 7 comments · Fixed by #17748, wazuh/wazuh-qa#4278 or wazuh/wazuh-documentation#6207
Assignees
@fdalmaup
Labels
level/task module/aws type/bug Something isn't working

Comments

@fdalmaup
Copy link
Member

fdalmaup commented Jun 2, 2023 •
edited
Loading

Affected integration
AWS

Description

We have found that the current implementation of the AWS Services integrations (Inspector and CloudWatch Logs) do not make use of the discard_regex functionality although an example of a configuration using it can be found in the documentation. The integrations take the value of the parameter when instantiated but they do not use it to filter the events it fetches because they lack the logic required for discard_regex to work. We should make the necessary changes to make the proper use of the field available.

Tasks

  • Implement the discard_regex feature for the services integrations.
  • Test in a manager.
  • Test in an agent.
  • Unit tests without failures. Updated if there are any relevant changes.
  • Integration tests without failures. Updated if there are any relevant changes.
  • Update the documentation if necessary.
  • Add entry to the changelog if necessary.
@Selutario Selutario added type/bug Something isn't working level/task labels Jun 7, 2023
@fdalmaup
Copy link
Member Author

Issue Update

Found that the methods used in the AWSBucket.iter_events method to apply the discard functionality can be refactored and added to the WazuhIntegration class in order to be inherited by `AWSService, thus reusing the code for buckets and services.

I have made the necessary modifications so the AWSInspector and AWSCloudWatchLogs classes implement the functionality. Currently in the process of making manual tests for code adjustments.

@fdalmaup
Copy link
Member Author

fdalmaup commented Jun 30, 2023 •
edited
Loading

Issue Update

Following the modifications previously mentioned, the AWSInspector class behavior did not present any inconveniences as the logs are JSON objects, and the discard logic is directly applied.

Respecting how the AWSCloudWatchLogs class will work is influenced by the format of the logs that are received. As the received string by the module is sent directly to Analysisd, for cases in which there are JSON-type objects in the log stream, the code was modified so that the discard-field and discard-regex values are searched for if applicable. Otherwise, it looks for the discard-regex pattern in the string and the discard-field parameter is not required.

The DEBUG level required to show certain messages was also modified from 2 to 3 due to the number of logs that are generated with them.

Between executions of the module, the aws_services.db file was removed to get logs always.

Inspector

Command output Debug level 2 
/var/ossec/wodles/aws/aws-s3 -sr inspector -d 2 -s 2022-JAN-26 -p dev -r us-east-1 --discard-field assetAttributes.agentId --discard-regex "i-instance"
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Listing findings starting from 2022-01-26 00:00:00
DEBUG: +++ There are no new events in the "us-east-1" region
Command output Debug level 3 
/var/ossec/wodles/aws/aws-s3 -sr inspector -d 3 -s 2022-JAN-26 -p dev -r us-east-1 --discard-field assetAttributes.agentId --discard-regex "i-instance"
DEBUG: +++ Debug mode on - Level: 3
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Listing findings starting from 2022-01-26 00:00:00
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 9 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 5 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 11 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 11 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 11 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 6 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 12 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ There are no new events in the "us-east-1" region

CloudWatch Logs

Discard simple text log

Command output Debug level 2 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 2 -s 2021-JUN-20 -p dev -r us-east-1 --discard-regex Test
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1624147200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37592581196964608882919025805905778821310252234493460480/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ Sent 1 events to Analysisd
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1624147200000" and end_time "None"
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1624147200000, 'end_time': 1688068269999}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB
Command output Debug level 3 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 3 -s 2021-JUN-20 -p dev -r us-east-1 --discard-regex Test
DEBUG: +++ Debug mode on - Level: 3
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1624147200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37592581196964608882919025805905778821310252234493460480/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: The message is "{     "networkInterfaces": [         {             "networkInterfaceId": "eni-networkInterfaceId",             "subnetId": "subnet-subnetId",             "vpcId": "vpc-vpcId",             "privateDnsName": "ip-0-0-0-0.ec2.internal",             "privateIpAddress": "0.0.0.0",             "privateIpAddresses": [                 {                     "privateDnsName": "ip-0-0-0-0.ec2.internal",                     "privateIpAddress": "0.0.0.0"                 }             ],             "publicDnsName": "ec2-0-0-0-0.compute-1.amazonaws.com",             "publicIp": "0.0.0.0",             "ipv6Addresses": [],             "securityGroups": [                 {                     "groupName": "groupName0",                     "groupId": "sg-groupId0"                 },                 {                     "groupName": "groupName1",                     "groupId": "sg-groupID1"                 }             ]         }     ] }"
DEBUG: The message's timestamp is 1688068269999
DEBUG: "{     \"networkInterfaces\": [         {             \"networkInterfaceId\": \"eni-networkInterfaceId\",             \"subnetId\": \"subnet-subnetId\",             \"vpcId\": \"vpc-vpcId\",             \"privateDnsName\": \"ip-0-0-0-0.ec2.internal\",             \"privateIpAddress\": \"0.0.0.0\",             \"privateIpAddresses\": [                 {                     \"privateDnsName\": \"ip-0-0-0-0.ec2.internal\",                     \"privateIpAddress\": \"0.0.0.0\"                 }             ],             \"publicDnsName\": \"ec2-0-0-0-0.compute-1.amazonaws.com\",             \"publicIp\": \"0.0.0.0\",             \"ipv6Addresses\": [],             \"securityGroups\": [                 {                     \"groupName\": \"groupName0\",                     \"groupId\": \"sg-groupId0\"                 },                 {                     \"groupName\": \"groupName1\",                     \"groupId\": \"sg-groupID1\"                 }             ]         }     ] }"
DEBUG: +++ Sent 1 events to Analysisd
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1624147200000" and end_time "None"
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1624147200000, 'end_time': 1688068269999}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB

Discard JSON log

Command output Debug level 2 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 2 -s 2023-JUN-20 -p dev -r us-east-1 --discard-field networkInterfaces.networkInterfaceId --discard-regex eni-networkInterfaceId
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1687219200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1687219200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1687219200000" and end_time "None"
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1687219200000, 'end_time': 1687219200000}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB
Command output Debug level 3 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 3 -s 2023-JUN-20 -p dev -r us-east-1 --discard-field networkInterfaces.networkInterfaceId --discard-regex eni-networkInterfaceId
DEBUG: +++ Debug mode on - Level: 3
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1687219200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1687219200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ The "eni-networkInterfaceId" regex found a match in the "networkInterfaces.networkInterfaceId" field. The event will be skipped.
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1687219200000" and end_time "None"
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1687219200000, 'end_time': 1687219200000}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB

Conclusion

The manual testing passed, what remains to be done is executing the module's integration tests and updating them accordingly.

@fdalmaup
Copy link
Member Author

fdalmaup commented Jul 3, 2023 •
edited
Loading

Issue Update

The AWS ITs have passed successfully:

/wazuh-qa/tests/integration/test_aws# pytest -x .
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /wazuh-qa/tests/integration, configfile: pytest.ini
plugins: testinfra-5.0.0, html-3.1.1, metadata-3.0.0
collected 195 items                                                                                                                                             

test_basic.py ................                                                                                                                            [  8%]
test_discard_regex.py ..............                                                                                                                      [ 15%]
test_log_groups.py ..                                                                                                                                     [ 16%]
test_only_logs_after.py .............................................x.                                                                                   [ 40%]
test_parser.py ..........................                                                                                                                 [ 53%]
test_path.py ..........................................                                                                                                   [ 75%]
test_path_suffix.py .........                                                                                                                             [ 80%]
test_regions.py ........................                                                                                                                  [ 92%]
test_remove_from_bucket.py ...sss.........                                                                                                                [100%]

==================================================== 191 passed, 3 skipped, 1 xfailed in 3893.63s (1:04:53) =====================================================

Nevertheless, there were no cases to check the discard functionality for the AWSServices (Inspector and CloudWatchLogs). These are being developed in the 17388-aws-services-discard-regex-its branch to maintain the ITs updated. The case for AWSInspector is already done and the cases for AWSCloudWatchLogs are in progress. Regarding the former, some modifications are needed in the module's parser to allow only the discard_regex parameter without a field for simple text cases.

Also, the debug level for the logs that match the regex has been reestablished to 2 to maintain consistency with the messages for bucket cases.

fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
refactor(wazuh/wazuh#17388):rename discard cases files
1899498
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
b61c0f0
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
3593ffa
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
2c98a06
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
f82b764
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
b56028c
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
ec1f14f
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
7395787
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
05606f7
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
b6fddfc
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
bf00419
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add cloudwatch discard regex test and cases
ee3af70
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
adef47c
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
b70b299
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
dda4e9c
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 3, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
7b7b00e
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
18cd27b
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
e0d8e22
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): add inspector discard regex test and cases
0032d7e
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): separate cloudwatch test cases
9acb505
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): separate cloudwatch test cases
64fede1
@fdalmaup
Copy link
Member Author

fdalmaup commented Jul 4, 2023 •
edited
Loading

Issue Update

The team has decided to modify the current behavior of the module. Currently, when the <discard_regex> parameter did not have the field parameter or if it was empty, the module showed a Warning when restarting the central components of the Wazuh manager. We are changing this in order to break the restart process and show an error message when no field value is present, except for when the type is cloudwatchlogs since it admits <discard_regex> when expecting text logs. For this last case, no warning is shown, assuming the user is aware of this use of the parameter having read the documentation (WIP)

The ITs for discard_regex are done and should be merged in wazuh/wazuh-qa#4278:

pytest -vv wazuh-qa/tests/integration/test_aws/test_discard_regex.py
====================================================================== test session starts ======================================================================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.10.6', 'Platform': 'Linux-5.19.0-1025-aws-x86_64-with-glibc2.35', 'Packages': {'pytest': '7.1.2', 'pluggy': '1.0.0'}, 'Plugins': {'testinfra': '5.0.0', 'html': '3.1.1', 'metadata': '3.0.0'}}
rootdir: /wazuh-qa/tests/integration, configfile: pytest.ini
plugins: testinfra-5.0.0, html-3.1.1, metadata-3.0.0
collected 17 items                                                                                                                                              

wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[cloudtrail_discard_regex] PASSED                                     [  5%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[vpc_discard_regex] PASSED                                            [ 11%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[config_discard_regex] PASSED                                         [ 17%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[alb_discard_regex] PASSED                                            [ 23%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[clb_discard_regex] PASSED                                            [ 29%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[nlb_discard_regex] PASSED                                            [ 35%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[kms_discard_regex] PASSED                                            [ 41%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[macie_discard_regex] PASSED                                          [ 47%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[trusted_advisor_discard_regex] PASSED                                [ 52%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[guardduty_discard_regex] PASSED                                      [ 58%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[native_guardduty_discard_regex] PASSED                               [ 64%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[waf_discard_regex] PASSED                                            [ 70%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[server_access_discard_regex] PASSED                                  [ 76%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_bucket_discard_regex[cisco_umbrella_discard_regex] PASSED                                 [ 82%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_cloudwatch_discard_regex_json[cloudwatch_discard_regex_json] PASSED                       [ 88%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_cloudwatch_discard_regex_simple_text[cloudwatch_discard_regex_simple_text] PASSED         [ 94%]
wazuh-qa/tests/integration/test_aws/test_discard_regex.py::test_inspector_discard_regex[inspector_discard_regex] PASSED                                   [100%]

================================================================ 17 passed in 336.47s (0:05:36) =================================================================

The documentation modifications are being added in wazuh/wazuh-documentation#6207.

fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): separate cloudwatch test cases
65e2972
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): separate cloudwatch test cases
d5a82f9
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): separate cloudwatch test cases
ac69afc
fdalmaup added a commit to wazuh/wazuh-qa that referenced this issue Jul 4, 2023
@fdalmaup
feat(wazuh/wazuh#17388): separate cloudwatch test cases
e50f774
This was referenced Jul 4, 2023
Merged
Merged
Merged
This was linked to pull requests Jul 5, 2023
Add AWSServices discard_regex integration tests wazuh/wazuh-qa#4278
Merged
Add discard_regex parameter to Services wazuh/wazuh-documentation#6207
Merged
@fdalmaup
Copy link
Member Author

fdalmaup commented Jul 12, 2023 •
edited
Loading

Issue Update

The feature is required for the upcoming 4.5.1 release, therefore a rebase to the corresponding branch was required. This release does not have the latest changes on which the module integration tests were based.
The IT modifications carried out in wazuh/wazuh-qa#4278 will stay pointing to the branch with the latest changes, but the development needed to be manually tested.

Inspector

Command output Debug level 2 
/var/ossec/wodles/aws/aws-s3 -sr inspector -d 2 -s 2022-JAN-26 -p dev -r us-east-1 --discard-field assetAttributes.agentId --discard-regex "i-instance"
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Listing findings starting from 2022-01-26 00:00:00
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ There are no new events in the "us-east-1" region
Command output Debug level 3 
/var/ossec/wodles/aws/aws-s3 -sr inspector -d 3 -s 2022-JAN-26 -p dev -r us-east-1 --discard-field assetAttributes.agentId --discard-regex "i-instance"
DEBUG: +++ Debug mode on - Level: 3
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Listing findings starting from 2022-01-26 00:00:00
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 9 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 5 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 11 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 11 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 11 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 6 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 12 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 8 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ Processing 4 events
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ The "i-instance" regex found a match in the "assetAttributes.agentId" field. The event will be skipped.
DEBUG: +++ There are no new events in the "us-east-1" region

CloudWatch Logs

Discard simple text log

Command output Debug level 2 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 2 -s 2021-JUN-20 -p dev -r us-east-1 --discard-regex Test
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1624147200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37592581196964608882919025805905778821310252234493460480/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: The message is "{     "networkInterfaces": [         {             "networkInterfaceId": "eni-networkInterfaceId",             "subnetId": "subnet-subnetId",             "vpcId": "vpc-vpcId",             "privateDnsName": "ip-0-0-0-0.ec2.internal",             "privateIpAddress": "0.0.0.0",             "privateIpAddresses": [                 {                     "privateDnsName": "ip-0-0-0-0.ec2.internal",                     "privateIpAddress": "0.0.0.0"                 }             ],             "publicDnsName": "ec2-0-0-0-0.compute-1.amazonaws.com",             "publicIp": "0.0.0.0",             "ipv6Addresses": [],             "securityGroups": [                 {                     "groupName": "groupName0",                     "groupId": "sg-groupId0"                 },                 {                     "groupName": "groupName1",                     "groupId": "sg-groupID1"                 }             ]         }     ] }"
DEBUG: +++ Sent 1 events to Analysisd
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1624147200000, 'end_time': 1688068269999}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB
Command output Debug level 3 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 3 -s 2021-JUN-20 -p dev -r us-east-1 --discard-regex Test
DEBUG: +++ Debug mode on - Level: 3
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1624147200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: +++ The "Test" regex found a match. The event will be skipped.
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37592581196964608882919025805905778821310252234493460480/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: The message is "{     "networkInterfaces": [         {             "networkInterfaceId": "eni-networkInterfaceId",             "subnetId": "subnet-subnetId",             "vpcId": "vpc-vpcId",             "privateDnsName": "ip-0-0-0-0.ec2.internal",             "privateIpAddress": "0.0.0.0",             "privateIpAddresses": [                 {                     "privateDnsName": "ip-0-0-0-0.ec2.internal",                     "privateIpAddress": "0.0.0.0"                 }             ],             "publicDnsName": "ec2-0-0-0-0.compute-1.amazonaws.com",             "publicIp": "0.0.0.0",             "ipv6Addresses": [],             "securityGroups": [                 {                     "groupName": "groupName0",                     "groupId": "sg-groupId0"                 },                 {                     "groupName": "groupName1",                     "groupId": "sg-groupID1"                 }             ]         }     ] }"
DEBUG: The message's timestamp is 1688068269999
DEBUG: "{     \"networkInterfaces\": [         {             \"networkInterfaceId\": \"eni-networkInterfaceId\",             \"subnetId\": \"subnet-subnetId\",             \"vpcId\": \"vpc-vpcId\",             \"privateDnsName\": \"ip-0-0-0-0.ec2.internal\",             \"privateIpAddress\": \"0.0.0.0\",             \"privateIpAddresses\": [                 {                     \"privateDnsName\": \"ip-0-0-0-0.ec2.internal\",                     \"privateIpAddress\": \"0.0.0.0\"                 }             ],             \"publicDnsName\": \"ec2-0-0-0-0.compute-1.amazonaws.com\",             \"publicIp\": \"0.0.0.0\",             \"ipv6Addresses\": [],             \"securityGroups\": [                 {                     \"groupName\": \"groupName0\",                     \"groupId\": \"sg-groupId0\"                 },                 {                     \"groupName\": \"groupName1\",                     \"groupId\": \"sg-groupID1\"                 }             ]         }     ] }"
DEBUG: +++ Sent 1 events to Analysisd
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1624147200000, 'end_time': 1688068269999}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB

Discard JSON log

Command output Debug level 2 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 2 -s 2021-JUN-20 -p dev -r us-east-1 --discard-field networkInterfaces.networkInterfaceId --discard-regex eni-networkInterfaceId
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1624147200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: The message is "Test log event 2022-07-26"
DEBUG: The message is "Test log 0 - 2022-7-29"
DEBUG: The message is "Test log 1 - 2022-7-29"
DEBUG: The message is "Test log 2 - 2022-7-29"
DEBUG: The message is "Test log 3 - 2022-7-29"
DEBUG: The message is "Test log 4 - 2022-7-29"
DEBUG: The message is "Test log 5 - 2022-7-29"
DEBUG: The message is "Test log 6 - 2022-7-29"
DEBUG: The message is "Test log 7 - 2022-7-29"
DEBUG: The message is "Test log 8 - 2022-7-29"
DEBUG: The message is "Test log 9 - 2022-7-29"
DEBUG: The message is "Test log 10 - 2022-7-29"
DEBUG: The message is "Test log 11 - 2022-7-29"
DEBUG: The message is "Test log 12 - 2022-7-29"
DEBUG: The message is "Test log 13 - 2022-8-1      1659355579"
DEBUG: The message is "Test log 14 - 2022-8-5   1659625348516"
DEBUG: The message is "Test log 15 - 2022-8-10    1660152255717"
DEBUG: The message is "Test log 16 - 2023-6-02 1685709634"
DEBUG: +++ Sent 18 events to Analysisd
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37592581196964608882919025805905778821310252234493460480/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ The "eni-networkInterfaceId" regex found a match in the "networkInterfaces.networkInterfaceId" field. The event will be skipped.
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1624147200000, 'end_time': 1685709641642}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB
Command output Debug level 3 
/var/ossec/wodles/aws/aws-s3 -sr cloudwatchlogs -g 4_5_test -d 3 -s 2021-JUN-20 -p dev -r us-east-1 --discard-field networkInterfaces.networkInterfaceId --discard-regex eni-networkInterfaceId
DEBUG: +++ Debug mode on - Level: 3
DEBUG: +++ Getting alerts from "us-east-1" region.
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: only logs: 1624147200000
DEBUG: +++ Table does not exist; create
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: Getting data from DB for log stream "test_stream" in log group "4_5_test"
DEBUG: Token: "None", start_time: "None", end_time: "None"
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "None", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log event 2022-07-26"
DEBUG: The message's timestamp is 1658862280721
DEBUG: "Test log event 2022-07-26"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 0 - 2022-7-29"
DEBUG: The message's timestamp is 1659100462660
DEBUG: "Test log 0 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 1 - 2022-7-29"
DEBUG: The message's timestamp is 1659100472502
DEBUG: "Test log 1 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 2 - 2022-7-29"
DEBUG: The message's timestamp is 1659100483581
DEBUG: "Test log 2 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 3 - 2022-7-29"
DEBUG: The message's timestamp is 1659100492369
DEBUG: "Test log 3 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 4 - 2022-7-29"
DEBUG: The message's timestamp is 1659100502933
DEBUG: "Test log 4 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 5 - 2022-7-29"
DEBUG: The message's timestamp is 1659100512690
DEBUG: "Test log 5 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 6 - 2022-7-29"
DEBUG: The message's timestamp is 1659100522240
DEBUG: "Test log 6 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 7 - 2022-7-29"
DEBUG: The message's timestamp is 1659100531170
DEBUG: "Test log 7 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 8 - 2022-7-29"
DEBUG: The message's timestamp is 1659100540310
DEBUG: "Test log 8 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 9 - 2022-7-29"
DEBUG: The message's timestamp is 1659100550044
DEBUG: "Test log 9 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 10 - 2022-7-29"
DEBUG: The message's timestamp is 1659100556663
DEBUG: "Test log 10 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 11 - 2022-7-29"
DEBUG: The message's timestamp is 1659100566406
DEBUG: "Test log 11 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 12 - 2022-7-29"
DEBUG: The message's timestamp is 1659100574840
DEBUG: "Test log 12 - 2022-7-29"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 13 - 2022-8-1      1659355579"
DEBUG: The message's timestamp is 1659355591835
DEBUG: "Test log 13 - 2022-8-1      1659355579"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 14 - 2022-8-5   1659625348516"
DEBUG: The message's timestamp is 1659625354421
DEBUG: "Test log 14 - 2022-8-5   1659625348516"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 15 - 2022-8-10    1660152255717"
DEBUG: The message's timestamp is 1660152262159
DEBUG: "Test log 15 - 2022-8-10    1660152255717"
DEBUG: +++ Retrieved log event is not a JSON object.
DEBUG: The message is "Test log 16 - 2023-6-02 1685709634"
DEBUG: The message's timestamp is 1685709641642
DEBUG: "Test log 16 - 2023-6-02 1685709634"
DEBUG: +++ Sent 18 events to Analysisd
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37592581196964608882919025805905778821310252234493460480/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ The "eni-networkInterfaceId" regex found a match in the "networkInterfaces.networkInterfaceId" field. The event will be skipped.
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Getting CloudWatch logs from log stream "test_stream" in log group "4_5_test" using token "f/37645180366972094805396383782547622777891308846091272192/s", start_time "1624147200000" and end_time "None"
DEBUG: +++ Sending events to Analysisd...
DEBUG: +++ There are no new events in the "4_5_test" group
DEBUG: Saving data for log group "4_5_test" and log stream "test_stream".
DEBUG: The saved values are "{'token': 'f/37645185496176941583196752952502274031084618732661604351/s', 'start_time': 1624147200000, 'end_time': 1685709641642}"
DEBUG: Purging the BD
DEBUG: Getting log streams for "4_5_test" log group
DEBUG: Found "test_stream" log stream in 4_5_test
DEBUG: committing changes and closing the DB

CloudTrail

Normal execution

ossec.conf 
<wodle name="aws-s3">
    <disabled>no</disabled>
    <remove_from_bucket>no</remove_from_bucket>
    <interval>1h</interval>
    <run_on_start>yes</run_on_start>
    <skip_on_error>no</skip_on_error>

    <bucket type="cloudtrail">
        <name>wazuh-aws-wodle-cloudtrail</name>
        <aws_profile>dev</aws_profile>
        <only_logs_after>2022-JAN-01</only_logs_after>
    </bucket>
</wodle>
ossec.log 
2023/07/12 15:51:36 wazuh-modulesd:aws-s3[3067] wm_aws.c:62 at wm_aws_main(): INFO: Module AWS started
2023/07/12 15:51:36 wazuh-modulesd:aws-s3[3067] wm_aws.c:84 at wm_aws_main(): INFO: Starting fetching of logs.
2023/07/12 15:51:36 wazuh-modulesd:aws-s3[3067] wm_aws.c:136 at wm_aws_main(): INFO: Executing Bucket Analysis: (Bucket: wazuh-aws-wodle-cloudtrail, Type: cloudtrail, Profile: dev)
2023/07/12 15:51:36 wazuh-modulesd:aws-s3[3067] wm_aws.c:376 at wm_aws_run_s3(): DEBUG: Create argument list
2023/07/12 15:51:36 wazuh-modulesd:aws-s3[3067] wm_aws.c:491 at wm_aws_run_s3(): DEBUG: Launching S3 Command: wodles/aws/aws-s3 --bucket wazuh-aws-wodle-cloudtrail --aws_profile dev --only_logs_after 2022-JAN-01 --type cloudtrail --debug 2
2023/07/12 15:51:41 wazuh-modulesd:aws-s3[3067] wm_aws.c:531 at wm_aws_run_s3(): DEBUG: Bucket:  -  OUTPUT: DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on 123456789123 - us-west-1
DEBUG: +++ Marker: AWSLogs/123456789123/CloudTrail/us-west-1/2022/01/01
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/01/25/123456789123_CloudTrail_us-west-1_20220125T0000Z_HASDOtJxgfdNInHa.json.gz
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/02/11/123456789123_CloudTrail_us-west-1_20220211T0000Z_HASDOtJxgfdNInHa.json.gz
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0000Z_HASDoKlxgfdNInHa.json.gz
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0000Z_HASDoKlxgfdNInHa.json.zip
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0000Z_HASDoKlxgfdkdIOa.json.txt
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0002Z_HASDoKlxgfdNInHa.json.zip
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/06/123456789123_CloudTrail_us-west-1_20230406T0002Z_HASDoKlxgfdNInHa.json
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/06/123456789123_CloudTrail_us-west-1_20230406T0002Z_HASDoKlxgfdNInHb.json
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/11/123456789123_CloudTrail_us-west-1_20230411T1755Z_2CqtXDyI0zFiYm1J.json
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/14/123456789123_CloudTrail_us-west-1_20230411T1755Z_2CqtXDyI0zFiYm1J.json
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on 789123123456 - us-west-1
DEBUG: +++ Marker: AWSLogs/789123123456/CloudTrail/us-west-1/2022/01/01
DEBUG: +++ No logs to process in bucket: 789123123456/us-west-1
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on 123123456789 - us-west-1
DEBUG: +++ Marker: AWSLogs/123123456789/CloudTrail/us-west-1/2022/01/01
DEBUG: ++ Found new log: AWSLogs/123123456789/CloudTrail/us-west-1/2023/01/25/123456789123_CloudTrail_us-west-1_20211223T0000Z_HASDOtJxxxxxxxxx.json.gz
DEBUG: +++ DB Maintenance

2023/07/12 15:51:41 wazuh-modulesd:aws-s3[3067] wm_aws.c:201 at wm_aws_main(): INFO: Fetching logs finished.
2023/07/12 15:51:41 wazuh-modulesd:aws-s3[3067] wm_aws.c:80 at wm_aws_main(): DEBUG: Sleeping until: 2023/07/12 16:51:36

discard_regex feature execution

ossec.conf 
<wodle name="aws-s3">
    <disabled>no</disabled>
    <remove_from_bucket>no</remove_from_bucket>
    <interval>1h</interval>
    <run_on_start>yes</run_on_start>
    <skip_on_error>no</skip_on_error>

    <bucket type="cloudtrail">
        <name>wazuh-aws-wodle-cloudtrail</name>
        <aws_profile>dev</aws_profile>
        <only_logs_after>2022-JAN-01</only_logs_after>
        <discard_regex field="eventSource">.*ec2.amazonaws.com.*</discard_regex>
    </bucket>
</wodle>
ossec.log 
2023/07/12 16:02:25 wazuh-modulesd:aws-s3[5016] wm_aws.c:62 at wm_aws_main(): INFO: Module AWS started
2023/07/12 16:02:25 wazuh-modulesd:aws-s3[5016] wm_aws.c:84 at wm_aws_main(): INFO: Starting fetching of logs.
2023/07/12 16:02:25 wazuh-modulesd:aws-s3[5016] wm_aws.c:136 at wm_aws_main(): INFO: Executing Bucket Analysis: (Bucket: wazuh-aws-wodle-cloudtrail, Type: cloudtrail, Profile: dev)
2023/07/12 16:02:25 wazuh-modulesd:aws-s3[5016] wm_aws.c:376 at wm_aws_run_s3(): DEBUG: Create argument list
2023/07/12 16:02:25 wazuh-modulesd:aws-s3[5016] wm_aws.c:491 at wm_aws_run_s3(): DEBUG: Launching S3 Command: wodles/aws/aws-s3 --bucket wazuh-aws-wodle-cloudtrail --aws_profile dev --only_logs_after 2022-JAN-01 --discard-field eventSource --discard-regex .*ec2.amazonaws.com.* --type cloudtrail --debug 2
2023/07/12 16:02:30 wazuh-modulesd:aws-s3[5016] wm_aws.c:531 at wm_aws_run_s3(): DEBUG: Bucket:  -  OUTPUT: DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on 123456789123 - us-west-1
DEBUG: +++ Marker: AWSLogs/123456789123/CloudTrail/us-west-1/2022/01/01
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/01/25/123456789123_CloudTrail_us-west-1_20220125T0000Z_HASDOtJxgfdNInHa.json.gz
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/02/11/123456789123_CloudTrail_us-west-1_20220211T0000Z_HASDOtJxgfdNInHa.json.gz
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0000Z_HASDoKlxgfdNInHa.json.gz
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0000Z_HASDoKlxgfdNInHa.json.zip
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0000Z_HASDoKlxgfdkdIOa.json.txt
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2022/03/30/123456789123_CloudTrail_us-west-1_20220330T0002Z_HASDoKlxgfdNInHa.json.zip
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/06/123456789123_CloudTrail_us-west-1_20230406T0002Z_HASDoKlxgfdNInHa.json
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/06/123456789123_CloudTrail_us-west-1_20230406T0002Z_HASDoKlxgfdNInHb.json
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/11/123456789123_CloudTrail_us-west-1_20230411T1755Z_2CqtXDyI0zFiYm1J.json
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: ++ Found new log: AWSLogs/123456789123/CloudTrail/us-west-1/2023/04/14/123456789123_CloudTrail_us-west-1_20230411T1755Z_2CqtXDyI0zFiYm1J.json
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ The ".*ec2.amazonaws.com.*" regex found a match in the "eventSource" field. The event will be skipped.
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on 789123123456 - us-west-1
DEBUG: +++ Marker: AWSLogs/789123123456/CloudTrail/us-west-1/2022/01/01
DEBUG: +++ No logs to process in bucket: 789123123456/us-west-1
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on 123123456789 - us-west-1
DEBUG: +++ Marker: AWSLogs/123123456789/CloudTrail/us-west-1/2022/01/01
DEBUG: ++ Found new log: AWSLogs/123123456789/CloudTrail/us-west-1/2023/01/25/123456789123_CloudTrail_us-west-1_20211223T0000Z_HASDOtJxxxxxxxxx.json.gz
DEBUG: +++ DB Maintenance

2023/07/12 16:02:30 wazuh-modulesd:aws-s3[5016] wm_aws.c:201 at wm_aws_main(): INFO: Fetching logs finished.
2023/07/12 16:02:30 wazuh-modulesd:aws-s3[5016] wm_aws.c:80 at wm_aws_main(): DEBUG: Sleeping until: 2023/07/12 17:02:25

Conclusion

The module was successfully modified and still works as expected. If no field attribute is specified in the module's configuration inside the ossec.conf, the Wazuh server does not start and shows an error message, except for the mentioned case for CloudWatch Logs.

@Selutario
Copy link
Contributor

The development is already merged:

The discard_regex has been added to 4.5.1 but AWS ITs were added in wazuh-qa 4.6.0, so the PR which adds new use cases for discard_regex in cloudwatchlogs and inspector can't be merged until 4.5.1 is merged again in 4.6.0:

Moving the issue to blocked until merging the IT is possible.

@Selutario
Copy link
Contributor

Closing issue to not exceed code delivery ETA. AWS IT PR (wazuh/wazuh-qa#4278) cannot be merged until the development done in #17748 is pushed to all other branches.

@Selutario Selutario mentioned this issue Aug 7, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fdalmaup fdalmaup

Labels
level/task module/aws type/bug Something isn't working
Projects
No open projects
Release 4.5.1
Status: Done
Milestone
No milestone
2 participants
@Selutario @fdalmaup