[filebeat][decode_cef] Unable to parse fields containing hyphen -

[kcreddy]

This is an extension to #40236 where a workaround was performed before decode_cef processor as it is unable to handle fields containing hyphen -.

Sample message:

2536 <14>1 2024-07-04T09:16:35.992Z logfwd20-251f92c6-abd9-4da9-a32f-ea60baed66ca-taskmanager-wx85p logforwarder - panwlogs - CEF:0|Palo Alto Networks|LF|2.0|TRAFFIC|end|3|dtz=UTC rt=Jul 04 2024 09:03:48 deviceExternalId=no-serial PanOSConfigVersion=10.2 start=Jul 04 2024 09:03:39 src=127.0.0.1 dst=0.0.0.0 sourceTranslatedAddress=0.0.0.0 destinationTranslatedAddress= cs1=intrazone-default cs1Label=Rule suser= duser= app=unknown-udp cs3=vsys1 cs3Label=VirtualLocation cs4=untrust cs4Label=FromZone cs5=untrust cs5Label=ToZone deviceInboundInterface=ethernet1/1 deviceOutboundInterface=ethernet1/1 cs6=Cortex Data Lake cs6Label=LogSetting cn1=574297 cn1Label=SessionID cnt=1 spt=13442 dpt=500 sourceTranslatedPort=0 destinationTranslatedPort=0 proto=udp act=allow PanOSBytes=82 out=82 in=0 cn2=1 cn2Label=PacketsTotal PanOSSessionStartTime=Jul 04 2024 09:03:05 cn3=0 cn3Label=SessionDuration cs2=any cs2Label=URLCategory externalId=7361339208201408573 PanOSSourceLocation=DE PanOSDestinationLocation=US PanOSPacketsSent=1 PanOSPacketsReceived=0 reason=aged-out PanOSDGHierarchyLevel1=65544 PanOSDGHierarchyLevel2=65545 PanOSDGHierarchyLevel3=65546 PanOSDGHierarchyLevel4=65550 PanOSVirtualSystemName= dvchost=GP cloud service cat=from-policy PanOSSourceUUID= PanOSDestinationUUID= PanOSIMSI=0 PanOSIMEI= PanOSParentSessionID=0 PanOSParentStarttime=Jan 01 1970 00:00:00 PanOSTunnel=N/A PanOSEndpointAssociationID=0 PanOSChunksTotal=0 PanOSChunksSent=0 PanOSChunksReceived=0 PanOSRuleUUID=40b8842f-eec3-4e04-b760-6a2ce4698fde PanOSHTTP2Connection=0 PanOSLinkChangeCount=0 PanOSSDWANPolicyName= PanOSLinkSwitches= PanOSSDWANCluster= PanOSSDWANDeviceType= PanOSSDWANClusterType= PanOSSDWANSite= PanOSDynamicUserGroupName= PanOSX-Forwarded-ForIP= PanOSSourceDeviceCategory= PanOSSourceDeviceProfile= PanOSSourceDeviceModel= PanOSSourceDeviceVendor= PanOSSourceDeviceOSFamily= PanOSSourceDeviceOSVersion= PanOSSourceDeviceHost= PanOSSourceDeviceMac= PanOSDestinationDeviceCategory= PanOSDestinationDeviceProfile= PanOSDestinationDeviceModel= PanOSDestinationDeviceVendor= PanOSDestinationDeviceOSFamily= PanOSDestinationDeviceOSVersion= PanOSDestinationDeviceHost= PanOSDestinationDeviceMac= PanOSContainerID= PanOSContainerNameSpace= PanOSContainerName= PanOSSourceEDL= PanOSDestinationEDL= PanOSGPHostID= PanOSEndpointSerialNumber= PanOSSourceDynamicAddressGroup= PanOSDestinationDynamicAddressGroup= PanOSHASessionOwner= PanOSTimeGeneratedHighResolution=Jul 04 2024 09:03:39 PanOSNSSAINetworkSliceType= PanOSNSSAINetworkSliceDifferentiator=

If decode_cef is applied to above message, we get error: malformed value for PanOSDynamicUserGroupName at pos 1617, because it is unable to parse adjacent field PanOSX-Forwarded-ForIP. When a workaround is applied to remove hyphen - from the field name, this error is resolved.
Below is the filebeat configuration with current workaround (removing hyphen - from fields) to mitigate the errors.

Filebeat configuration:

filebeat.inputs:
- type: tcp
  host: "127.0.0.1:9528"
  processors:
  - copy_fields:
      fields:
        - from: message
          to: event.original
      fail_on_error: true
      ignore_missing: false
  - replace:
      fields:
        - field: "message"
          pattern: "PanOSX-Forwarded-ForIP="
          replacement: "PanOSXForwardedForIP="
        - field: "message"
          pattern: "PanOSX-Forwarded-For="
          replacement: "PanOSXForwardedFor="
        - field: "message"
          pattern: "PanOSSplit-tunnelconfiguration="
          replacement: "PanOSSplitTunnelconfiguration="
      ignore_missing: false
      fail_on_error: true
  - decode_cef:
      field: message
      ecs: false
  - drop_fields:
      fields: ["message"]
      ignore_missing: false

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[andrewkroh]

The spec says extensions are alphanumeric.

But we made an exception already so allowing - won't be any bigger of a deviation IMO.

beats/x-pack/filebeat/processors/decode_cef/cef/cef.rl

Line 35 in f1df291

extension_key_chars = extension_key_start_chars | '.' | ',' | '[' | ']';