Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

M2: ASIM Parsers, connector optimizations/enhancements #10994

Merged
merged 62 commits into from
Oct 17, 2024
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
e346896
added support to filter flow summaries by type and refactored code
ashwinvenkatesha Aug 16, 2024
cf360eb
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Aug 20, 2024
61f32c4
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Aug 22, 2024
0cfac85
added asim parsers for audit and network session schema and added the…
ashwinvenkatesha Aug 22, 2024
8eed458
added SN to flow events schema
ashwinvenkatesha Aug 22, 2024
f2bb73b
update network session parser
ashwinvenkatesha Aug 22, 2024
56781b3
remove type, tenantid, eventdetails from network session asim
ashwinvenkatesha Sep 16, 2024
2388e1e
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Sep 16, 2024
207524e
handle missing comma
ashwinvenkatesha Sep 16, 2024
3501c2b
fix non-ascii chars, fix asim query for paramter and parameterless
ashwinvenkatesha Sep 17, 2024
549981f
fix formatting
ashwinvenkatesha Sep 17, 2024
a4b0f76
fix indentation and remove trailing whitespaces
ashwinvenkatesha Sep 17, 2024
1fbaecc
remove extend usage and simplify query
ashwinvenkatesha Sep 17, 2024
21c75be
remove semicolon at end of query
ashwinvenkatesha Sep 17, 2024
ac811cd
fix bugs
ashwinvenkatesha Sep 18, 2024
6918ae4
fix a bug, add sample data for asim parser, add schema test file for …
ashwinvenkatesha Sep 18, 2024
18630f6
adjust asim queries for audit and network session based on tests and …
ashwinvenkatesha Sep 19, 2024
646a9b3
fix errors
ashwinvenkatesha Sep 19, 2024
0a200ac
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Sep 19, 2024
2fd8c74
remove extra comma from asim query
ashwinvenkatesha Sep 19, 2024
e220d5f
remove extra file
ashwinvenkatesha Sep 19, 2024
1f78071
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Sep 23, 2024
dadded0
address template validation errors by renaming ingestedlogs.csv and s…
ashwinvenkatesha Sep 23, 2024
76802ed
add custom table schema to fix data ingestion test failure
ashwinvenkatesha Sep 23, 2024
2743433
fix test files for network schema asim parser
ashwinvenkatesha Sep 23, 2024
de66aaa
include asim_ingested logs.csv, fix indentation
ashwinvenkatesha Sep 23, 2024
b95f6ff
rename schema files
ashwinvenkatesha Sep 23, 2024
258d076
fix typo
ashwinvenkatesha Sep 23, 2024
68041ba
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Sep 25, 2024
8b48904
update parsers to remove tenantid, icmp_type; set usernametype in net…
ashwinvenkatesha Sep 25, 2024
63d51ef
update table schema for kqlvalidation tests
ashwinvenkatesha Sep 25, 2024
4ce7e9f
set ipaddr to empty string if not found vs unknown; update sameple data
ashwinvenkatesha Sep 25, 2024
79089d5
fix actorusernametype and hostname_has_any filter
ashwinvenkatesha Sep 25, 2024
e71ea6c
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 1, 2024
d1c6db0
added auth parser, ingested logs.csv, schema and data test results
ashwinvenkatesha Oct 4, 2024
f129795
address comments
ashwinvenkatesha Oct 4, 2024
6b3f638
fix merge conflict
ashwinvenkatesha Oct 4, 2024
fbbd320
comment out _ItemId usage, fix typos, rename ingestedlogs.csv based o…
ashwinvenkatesha Oct 4, 2024
33b5b3a
fix kql validation errors, add additional ingestedlogs.csv
ashwinvenkatesha Oct 4, 2024
63d7466
fix TYPE of auth events
ashwinvenkatesha Oct 4, 2024
716a6ae
minor changes
ashwinvenkatesha Oct 4, 2024
9adb9e9
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 7, 2024
2b9ad64
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 8, 2024
93c3eef
fix sample data
ashwinvenkatesha Oct 8, 2024
fba4ddd
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 9, 2024
1da64bd
add _Itemid to kqlvalidation schema; add product details to asimteste…
ashwinvenkatesha Oct 9, 2024
c1f4851
fix kqlvalidation failure
ashwinvenkatesha Oct 9, 2024
8f5f1b0
fix another kqlvalidation error
ashwinvenkatesha Oct 9, 2024
fc71e73
updated createuidefn, azuredeploy, constants.py, aws_queuy.py
ashwinvenkatesha Oct 10, 2024
6dc9a75
fix ingested logs.csv
ashwinvenkatesha Oct 10, 2024
5219489
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 10, 2024
cf89a5d
change default value to array
ashwinvenkatesha Oct 10, 2024
5ae7867
remove array parameter from azuredeploy
ashwinvenkatesha Oct 10, 2024
359ba25
fix file ref
ashwinvenkatesha Oct 10, 2024
268a775
fix aws_queue.py and update zip
ashwinvenkatesha Oct 11, 2024
f19f249
fix merge conflicts
ashwinvenkatesha Oct 14, 2024
650e365
repackaged solution, use consistent filtering in vimAuth vs ASimAuth …
ashwinvenkatesha Oct 14, 2024
f94d7bd
add networkTrafficLogTypes to func app env var
ashwinvenkatesha Oct 14, 2024
513580a
update package
ashwinvenkatesha Oct 14, 2024
1a5d2a3
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 16, 2024
f50c361
updated zip file, function app code
ashwinvenkatesha Oct 16, 2024
726c950
Merge remote-tracking branch 'origin/master' into sentinel_m2
ashwinvenkatesha Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,14 @@
{
"name": "version",
"type": "int"
},
{
"name": "TenantId",
"type": "String"
},
{
"Name": "_ItemId",
"Type": "String"
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,10 @@
"name": "un",
"type": "string"
},
{
"name": "sn",
"type": "string"
},
{
"name": "src_ip",
"type": "string"
Expand Down Expand Up @@ -128,6 +132,18 @@
{
"name": "version",
"type": "int"
}
},
{
"name": "icmp_type",
"type": "int"
},
{
"name": "TenantId",
"type": "String"
},
{
"Name": "_ItemId",
"Type": "String"
}
]
}
12 changes: 6 additions & 6 deletions ASIM/dev/ASimTester/ASimTester.csv
Original file line number Diff line number Diff line change
Expand Up @@ -547,13 +547,13 @@ EventOwner,string,Optional,ProcessEvent,,,
EventOwner,string,Optional,RegistryEvent,,,
EventOwner,string,Optional,UserManagement,,,
EventOwner,string,Optional,WebSession,,,
EventProduct,string,Mandatory,Authentication,Enumerated,Service Cloud|Auth0|CloudTrail|AAD|ASA|Microsoft Defender for IoT|ISE|M365 Defender for Endpoint|Meraki|Security Events|Okta|PostgreSQL|OpenSSH|su|sudo|Vectra XDR|SentinelOne|WAF|FalconHost|Carbon Black Cloud|Cortex Data Lake|Workspace,
EventProduct,string,Mandatory,AuditEvent,Enumerated,Azure|WAF|Security Events|Exchange 365|Dataminr Pulse|ISE|XDR|Meraki|FalconHost|SentinelOne|Carbon Black Cloud|BloxOne,
EventProduct,string,Mandatory,Authentication,Enumerated,Service Cloud|Auth0|CloudTrail|AAD|ASA|Microsoft Defender for IoT|ISE|M365 Defender for Endpoint|Meraki|Security Events|Okta|PostgreSQL|OpenSSH|su|sudo|Vectra XDR|SentinelOne|WAF|FalconHost|Carbon Black Cloud|Cortex Data Lake|Workspace|Core,
EventProduct,string,Mandatory,AuditEvent,Enumerated,Azure|WAF|Security Events|Exchange 365|Dataminr Pulse|ISE|XDR|Meraki|FalconHost|SentinelOne|Carbon Black Cloud|BloxOne|Core,
EventProduct,string,Mandatory,Common,,,
EventProduct,string,Mandatory,DhcpEvent,,BloxOne,
EventProduct,string,Mandatory,FileEvent,Enumerated,Security Events|Sysmon for Linux|Sysmon|M365 Defender for Endpoint|Azure File Storage|SharePoint|OneDrive|SentinelOne|Carbon Black Cloud|Workspace,
EventProduct,string,Mandatory,Dns,Enumerated,Umbrella|Azure Firewall|DNS Server|Sysmon|Sysmon for Linux|ZIA DNS|NIOS|Cloud DNS|Zeek|Vectra Stream|SentinelOne|FortiGate|BloxOne,
EventProduct,string,Mandatory,NetworkSession,Enumerated,Fortigate|IOS|ISE|SDP|Vectra Stream|NSGFlow|Fireware|VPC|Azure Defender for IoT|Azure Firewall|M365 Defender for Endpoint|Sysmon|Sysmon for Linux|Windows Firewall|WireData|ZIA Firewall|CDL|PanOS|VMConnection|Meraki|Zeek|Firewall|ASA|Cynerio|SentinelOne|WAF|Firepower|FalconHost|Carbon Black Cloud|Cortex Data Lake,
EventProduct,string,Mandatory,NetworkSession,Enumerated,Fortigate|IOS|ISE|SDP|Vectra Stream|NSGFlow|Fireware|VPC|Azure Defender for IoT|Azure Firewall|M365 Defender for Endpoint|Sysmon|Sysmon for Linux|Windows Firewall|WireData|ZIA Firewall|CDL|PanOS|VMConnection|Meraki|Zeek|Firewall|ASA|Cynerio|SentinelOne|WAF|Firepower|FalconHost|Carbon Black Cloud|Cortex Data Lake|Core,
EventProduct,string,Mandatory,ProcessEvent,Enumerated,M365 Defender for Endpoint|Sysmon for Linux|Sysmon|Azure Defender for IoT|Security Events|SentinelOne|Carbon Black Cloud|Vision One,
EventProduct,string,Mandatory,RegistryEvent,Enumerated,M365 Defender for Endpoint|Security Events|Sysmon|Windows Event|SentinelOne|Carbon Black Cloud|Vision One,
EventProduct,string,Mandatory,WebSession,Enumerated,IIS|Squid Proxy|ZIA Proxy|Vectra Stream|PanOS|CDL|Fireware|Meraki|Web Security Gateway|Zeek|Dataminr Pulse|HTTP Server|Fortigate|WAF|ASM|NetScaler|Firepower|Cortex Data Lake|Firewall,
Expand Down Expand Up @@ -677,13 +677,13 @@ EventUid,string,Recommended,ProcessEvent,,,
EventUid,string,Recommended,RegistryEvent,,,
EventUid,string,Recommended,UserManagement,,,
EventUid,string,Recommended,WebSession,,,
EventVendor,string,Mandatory,Authentication,Enumerated,Salesforce|AWS|Barracuda|Cisco|Microsoft|Okta|PostgreSQL|OpenBSD|Linux|Vectra|SentinelOne|CrowdStrike|VMware|Google,
EventVendor,string,Mandatory,AuditEvent,Enumerated,Microsoft|AWS|Barracuda|Cisco|Dataminr|Vectra|CrowdStrike|SentinelOne|VMware|Infoblox,
EventVendor,string,Mandatory,Authentication,Enumerated,Salesforce|AWS|Barracuda|Cisco|Microsoft|Okta|PostgreSQL|OpenBSD|Linux|Vectra|SentinelOne|CrowdStrike|VMware|Google|Illumio,
EventVendor,string,Mandatory,AuditEvent,Enumerated,Microsoft|AWS|Barracuda|Cisco|Dataminr|Vectra|CrowdStrike|SentinelOne|VMware|Infoblox|Illumio,
EventVendor,string,Mandatory,Common,,,
EventVendor,string,Mandatory,DhcpEvent,,Infoblox,
EventVendor,string,Mandatory,FileEvent,Enumerated,Microsoft|SentinelOne|VMware|Google,
EventVendor,string,Mandatory,Dns,Enumerated,Cisco|Corelight|GCP|Infoblox|Microsoft|Zscaler|Vectra AI|SentinelOne|Fortinet,
EventVendor,string,Mandatory,NetworkSession,Enumerated,Fortinet|AppGate|Barracuda|Palo Alto|Microsoft|Zscaler|AWS|Vectra AI|WatchGuard|Cisco|Corelight|Check Point|Forcepoint|Cynerio|SentinelOne|CrowdStrike|VMware|SonicWall,
EventVendor,string,Mandatory,NetworkSession,Enumerated,Fortinet|AppGate|Barracuda|Palo Alto|Microsoft|Zscaler|AWS|Vectra AI|WatchGuard|Cisco|Corelight|Check Point|Forcepoint|Cynerio|SentinelOne|CrowdStrike|VMware|SonicWall|Illumio,
EventVendor,string,Mandatory,ProcessEvent,Enumerated,Microsoft|SentinelOne|VMware|TrendMicro,
EventVendor,string,Mandatory,WebSession,Enumerated,Apache|Barracuda|Fortinet|Microsoft|Squid|Zscaler|Vectra AI|Palo Alto|WatchGuard|Cisco|Forcepoint|Corelight|Dataminr|Citrix|F5|SonicWall,
EventVendor,string,Mandatory,UserManagement,Enumerated,Microsoft|Linux|Cisco|SentinelOne,
Expand Down
4 changes: 3 additions & 1 deletion Parsers/ASimAuditEvent/Parsers/ASimAuditEvent.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Parsers:
- _ASim_AuditEvent_SentinelOne
ashwinvenkatesha marked this conversation as resolved.
Show resolved Hide resolved
- _ASim_AuditEvent_VMwareCarbonBlackCloud
- _ASim_AuditEvent_InfobloxBloxOne
- _ASim_AuditEvent_IllumioSaaSCore
ParserParams:
- Name: pack
Type: bool
Expand All @@ -56,5 +57,6 @@ ParserQuery: |
ASimAuditEventSentinelOne (BuiltInDisabled or ('ExcludeASimAuditEventSentinelOne' in (DisabledParsers))),
ASimAuditEventCrowdStrikeFalconHost(BuiltInDisabled or ('ExcludeASimAuditEventCrowdStrikeFalconHost' in (DisabledParsers))),
ASimAuditEventVMwareCarbonBlackCloud(BuiltInDisabled or ('ExcludeASimAuditEventVMwareCarbonBlackCloud' in (DisabledParsers))),
ASimAuditEventInfobloxBloxOne(BuiltInDisabled or ('ExcludeASimAuditEventInfobloxBloxOne' in (DisabledParsers)))
ASimAuditEventInfobloxBloxOne(BuiltInDisabled or ('ExcludeASimAuditEventInfobloxBloxOne' in (DisabledParsers))),
ASimAuditEventIllumioSaaSCore(BuiltInDisabled or ('ExcludeASimAuditEventIllumioSaaSCore' in (DisabledParsers)))
Loading
Loading