[cisco_meraki] Replace rsa2elk pipeline and add webhook input (#2897)

* Update data stream and pipeline for webhook based integration
* Add support for syslog (log) and webhook (events)
  - Log: Add pipeline tests for syslog flow, ipflow logs; Add sample_events for webhook system tests
  - Log: Add support for airmarshal events
  - Log: Add support for security_event ids_alerted
  - Log: Add support for security_event security_filtering_file_scanned type
  - Log: Add support for security_event security_filtering_disposition_change
  - Log: Add support for site-to-site vpn event type
  - Log: Anonymise test data
  - Log: Add support for vpn_connectivity_change and refactor field groups
  - Log: Add support for dhcp lease and no offers
  - Log: Add support for client_vpn_connect log messages
  - Log: Add support for urls type
  - Log: Add pipeline for ids-alerts
  - Log: Add system tests
  - Log: Update README; Disable webhook events by default; only enable syslog UDP by default
  - Log: Script to create event.category, type and action
  - Log: Add dashboards and screenshots
  - Log: Convert timestamp to date format in threat.indicator.last_seen
  - Log: Change dissect to grok for airmarshal events
  - Events: pipeline for top-level fields and event handling
  - Events: support for alert types
  - Events: support for all documented event types
  - Logs: Fixes, add support for subtypes
  - Logs: Add support for dfs_event, 8021x_auth, multiple_dhcp_servers_detected
  - Logs: use templating to avoid copying code

Co-authored-by: Andrew Kroh <[email protected]>

packages/cisco_meraki/_dev/build/docs/README.md

@@ -1,16 +1,64 @@
 # Cisco Meraki Integration
 
-This integration is for Cisco Meraki device's logs. It includes the following
-datasets for receiving logs over syslog or read from a file:
+Cisco Meraki offers a centralized cloud management platform for all Meraki devices such as MX Security Appliances, MR Access Points and so on. Its out-of-band cloud architecture creates secure, scalable and easy-to-deploy networks that can be managed from anywhere. This can be done from almost any device using web-based Meraki Dashboard and Meraki Mobile App. Each Meraki network generates its own events.
 
-- `log` dataset: supports Cisco Meraki logs.
+Cisco Meraki offers [several methods for device reporting](https://documentation.meraki.com/General_Administration/Monitoring_and_Reporting/Meraki_Device_Reporting_-_Syslog%2C_SNMP%2C_and_API). This integration supports gathering events via the Cisco Meraki syslog and via API reporting webhooks. The integration package allows you to search, observe, and visualize the events through Elasticsearch. 
+
+## Compatibility
+
+A syslog server can be configured to store messages for reporting purposes from MX Security Appliances, MR Access Points, and MS switches. This package collects events from the configured syslog server. The integration supports collection of events from "MX Security Appliances" and "MR Access Points". The "MS Switch" events are not recognized.
+
+## Configuration
+
+### Enabling the integration in Elastic
+
+1. In Kibana go to **Management > Integrations**
+2. In "Search for integrations" search bar type **Meraki**
+3. Click on "Cisco Meraki" integration from the search results.
+4. Click on **Add Cisco Meraki Integration** button to add the integration.
+
+### Cisco Meraki Dashboard Configuration
+
+#### Syslog
+
+Cisco Meraki dashboard can be used to configure one or more syslog servers and Meraki message types to be sent to the syslog servers. Refer to [Syslog Server Overview and Configuration](https://documentation.meraki.com/General_Administration/Monitoring_and_Reporting/Syslog_Server_Overview_and_Configuration#Configuring_a_Syslog_Server) page for more information on how to configure syslog server on Cisco Meraki.
+
+#### API Endpoint (Webhooks)
+
+Cisco Meraki dashboard can be used to configure Meraki webhooks. Refer to the [Webhooks Dashboard Setup](https://documentation.meraki.com/General_Administration/Monitoring_and_Reporting/Meraki_Device_Reporting_-_Syslog%2C_SNMP%2C_and_API#Webhooks_Dashboard_Setup) section.
+
+### Configure the Cisco Meraki integration
+
+#### Syslog
+
+Depending on the syslog server setup in your environment check one/more of the following options "Collect syslog from Cisco Meraki via UDP", "Collect syslog from Cisco Meraki via TCP", "Collect syslog from Cisco Meraki via file".
+
+Enter the values for syslog host and port OR file path based on the chosen configuration options.
+
+### API Endpoint (Webhooks)
+
+Check the option "Collect events from Cisco Meraki via Webhooks" option.
+
+1. Enter values for "Listen Address", "Listen Port" and "Webhook path" to form the endpoint URL. Make note of the **Endpoint URL** https://{AGENT_ADDRESS}:8686/meraki/events.
+2. Enter value for "Secret value". This must match the "Shared Secret" value entered when configuring the webhook from Meraki cloud.
+3. Enter values for "TLS". Cisco Meraki requires that the webhook accept requests over HTTPS. So you must either configure the integration with a valid TLS certificate or use a reverse proxy in front of the integration.
+
+### Log Events
+
+Enable to collect Cisco Meraki log events for all the applications configured for the chosen log stream.
 
 ## Logs
 
-### Meraki
+### Syslog
+
+The `cisco_meraki.log` dataset provides events from the configured syslog server. All Cisco Meraki syslog specific fields are available in the `cisco_meraki.log` field group.
 
-The `log` dataset collects Cisco Meraki logs.
+{{fields "log"}}
 
 {{event "log"}}
 
-{{fields "log"}}
+### API Endpoint (Webhooks)
+
+{{fields "events"}}
+
+{{event "events"}}

packages/cisco_meraki/_dev/deploy/docker/docker-compose.yml

@@ -1,18 +1,37 @@
-version: "2.3"
+version: '2.3'
 services:
- cisco-logfile:
+ meraki-webhook-http:
+ image: docker.elastic.co/observability/stream:v0.6.2
+ volumes:
+ - ./sample_events:/sample_events:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=http://elastic-agent:8686/meraki/events
+ - STREAM_WEBHOOK_HEADER=Authorization=abc123
+ command: log --start-signal=SIGHUP --delay=5s /sample_events/meraki-mx-ndjson.log
+ meraki-webhook-https:
+ image: docker.elastic.co/observability/stream:v0.6.2
+ volumes:
+ - ./sample_events:/sample_events:ro
+ environment:
+ - STREAM_PROTOCOL=webhook
+ - STREAM_ADDR=https://elastic-agent:8686/meraki/events
+ - STREAM_WEBHOOK_HEADER=Authorization=abc123
+ - STREAM_INSECURE=true
+ command: log --start-signal=SIGHUP --delay=5s /sample_events/meraki-mx-ndjson.log
+ cisco_meraki-log-logfile:
  image: alpine
  volumes:
  - ./sample_logs:/sample_logs:ro
  - ${SERVICE_LOGS_DIR}:/var/log
  command: /bin/sh -c "cp /sample_logs/* /var/log/"
- cisco-meraki-udp:
- image: docker.elastic.co/observability/stream:v0.5.0
+ cisco_meraki-log-udp:
+ image: docker.elastic.co/observability/stream:v0.6.2
  volumes:
  - ./sample_logs:/sample_logs:ro
- command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9514 -p=udp /sample_logs/cisco-meraki.log
- cisco-meraki-tcp:
- image: docker.elastic.co/observability/stream:v0.5.0
+ command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:8685 -p=udp /sample_logs/cisco-meraki.log
+ cisco_meraki-log-tcp:
+ image: docker.elastic.co/observability/stream:v0.6.2
  volumes:
  - ./sample_logs:/sample_logs:ro
- command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9514 -p=tcp /sample_logs/cisco-meraki.log
+ command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:8685 -p=tcp /sample_logs/cisco-meraki.log