Merge branch 'main' into new_fileevent_fields

.github/workflows/docs-preview-comment.yml

@@ -10,7 +10,7 @@ jobs:
  doc-preview:
  runs-on: ubuntu-latest
  steps:
- - uses: actions/github-script@v6
+ - uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
  name: Add doc preview links
  with:
  script: |

.github/workflows/stale.yml

@@ -15,10 +15,21 @@ jobs:
 
  steps:
  - name: "Check PRs"
- uses: actions/stale@v4
+ uses: actions/stale@28ca1036281a5e5922ead5184a1bbf96e5fc984e # v9
  with:
  repo-token: ${{ secrets.GITHUB_TOKEN }}
- stale-pr-message: 'This PR is stale because it has been open for 60 days with no activity.'
+ stale-pr-message: |
+ Hi!
+
+ We just realized that we haven't looked into this PR in a while. We're
+ sorry!
+
+ We're labeling this PR as `Stale` to make it hit our filters and
+ make sure we get back to it as soon as possible. In the meantime, it'd
+ be extremely helpful if you could take a look at it as well and confirm its
+ relevance. A simple comment with a nice emoji will be enough `:+1`.
+
+ Thank you for your contribution!
  stale-pr-label: 'stale'
  ascending: true
  days-before-pr-stale: 60

.github/workflows/test.yml

@@ -4,11 +4,11 @@ on: [push, pull_request]
 
 jobs:
  tests:
- runs-on: ubuntu-20.04
+ runs-on: ubuntu-22.04
  name: Unit Tests
  steps:
- - uses: actions/checkout@v2
- - uses: actions/setup-python@v2
+ - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4
+ - uses: actions/setup-python@39cd14951b08e74b54015e9e001cdefcf80e669f # v5
  with:
  python-version: '3.x'
  - run: git fetch --prune --unshallow --tags

renovate.json

@@ -0,0 +1,6 @@
+{
+ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
+ "extends": [
+ "local>elastic/renovate-config"
+ ]
+}

rfcs/text/0044-add-apple-platform-specific-fields.md

@@ -0,0 +1,119 @@
+# 0044: Apple Platform specific fields
+<!-- Leave this ID at 0000. The ECS team will assign a unique, contiguous RFC number upon merging the initial stage of this RFC. -->
+
+- Stage: **0 (strawperson)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
+- Date: **2024-08-13** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->
+
+
+### Summary
+This RFC proposes the addition of Apple platform-specific fields to the ECS schema. This enhancement will enable security software vendors to more accurately map out data, particularly for Apple platforms.
+
+The following feelds needs to be considered being added:
+
+## Fields
+
+##### Proposed New Fields for Process object
+
+Field | Type | Example | Description
+--- | --- | --- | ---
+responsible | keyword | Terminal.app | The responsible process on macOS, from an ancestry perspective, is the process that originally launched or spawned a given process.
+platform_binary | boolean | true | Indicates wethether this process executable is a default platform binary shipped with the operating system.
+endpoint_security_client | boolean | true | Indicates wethether this process executable is an Endpoint Security client.
+
+##### Proposed New Fields for Code Signature object
+
+Field | Type | Example | Description
+--- | --- | --- | ---
+flags | string | 570522385 | The flags used to sign the process.
+
+##### Proposed New Fields for Hash object
+
+Field | Type | Example | Description
+--- | --- | --- | ---
+cdhash | keyword | 3783b4052fd474dbe30676b45c329e7a6d44acd9 | The Code Directory (CD) hash of an executable
+
+##### Proposed New Fields for Device object
+
+Field | Type | Example | Description
+--- | --- | --- | ---
+serial_number | keyword | DJGAQS4CW5 | The unique serial number serves as a distinct identifier for each device, aiding in inventory management and device authentication.
+
+### Motivation
+
+As the number of Apple endpoints in enterprises grows, having the right fields to map data becomes increasingly valuable. This enables security researchers using Elastic, particularly those focusing on macOS, to query data more effectively by leveraging enriched data sets.
+
+## Usage
+
+As a developer at Jamf, working on the Elastic integration for Jamf Protect, our goal is to map as many fields as possible, especially as Jamf specializes in Apple platform security. While developing the integration, we've identified some gaps related to mapping events to ECS.
+
+These new fields offer versatile methods. For instance, they facilitate querying process executions by platform binaries or endpoint security clients without requiring specific identifiers. The added hash fields are particularly valuable for tracking the hash of an application bundle alongside the hash of the executable in the directory itself, while the others are self-explanatory.
+
+## Source data
+
+This data originates from Endpoint Security software operating on a macOS host and can be transmitted through various methods, including an Elastic Agent and as example the use of the Jamf Protect integration, which supports AWS S3 or HTTPs.
+
+<!--
+Stage 2: Included a real world example source document. Ideally this example comes from the source(s) identified in stage 1. If not, it should replace them. The goal here is to validate the utility of these field changes in the context of a real world example. Format with the source name as a ### header and the example document in a GitHub code block with json formatting, or if on the larger side, add them to the corresponding RFC folder.
+-->
+
+<!--
+Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2.
+-->
+
+<!--## Scope of impact
+
+<!--
+Stage 2: Identifies scope of impact of changes. Are breaking changes required? Should deprecation strategies be adopted? Will significant refactoring be involved? Break the impact down into:
+ * Ingestion mechanisms (e.g. beats/logstash)
+ * Usage mechanisms (e.g. Kibana applications, detections)
+ * ECS project (e.g. docs, tooling)
+The goal here is to research and understand the impact of these changes on users in the community and development teams across Elastic. 2-5 sentences each.
+-->
+
+<!--## Concerns
+
+<!--
+Stage 1: Identify potential concerns, implementation challenges, or complexity. Spend some time on this. Play devil's advocate. Try to identify the sort of non-obvious challenges that tend to surface later. The goal here is to surface risks early, allow everyone the time to work through them, and ultimately document resolution for posterity's sake.
+-->
+
+<!--
+Stage 2: Document new concerns or resolutions to previously listed concerns. It's not critical that all concerns have resolutions at this point, but it would be helpful if resolutions were taking shape for the most significant concerns.
+-->
+
+<!--
+Stage 3: Document resolutions for all existing concerns. Any new concerns should be documented along with their resolution. The goal here is to eliminate risk of churn and instability by ensuring all concerns have been addressed.
+-->
+
+## People
+
+The following are the people that consulted on the contents of this RFC.
+
+* txhaflaire | author
+
+<!--
+Who will be or has been consulted on the contents of this RFC? Identify authorship and sponsorship, and optionally identify the nature of involvement of others. Link to GitHub aliases where possible. This list will likely change or grow stage after stage.
+
+e.g.:
+
+* @Yasmina | author
+* @Monique | sponsor
+* @EunJung | subject matter expert
+* @JaneDoe | grammar, spelling, prose
+* @Mariana
+-->
+
+
+## References
+
+<!-- Insert any links appropriate to this RFC in this section. -->
+
+### RFC Pull Requests
+
+<!-- An RFC should link to the PRs for each of it stage advancements. -->
+
+* Stage 0: https:/elastic/ecs/pull/2338
+
+<!--
+* Stage 1: https:/elastic/ecs/pull/NNN
+...
+-->

rfcs/text/0044/code_signature.yml

@@ -0,0 +1,10 @@
+---
+- name: code_signature
+ fields:
+ - name: flags
+ level: extended
+ type: string
+ short: Code signing flags of the process
+ description: >
+ The flags used to sign the process.
+ example: 570522385

rfcs/text/0044/device.yml

@@ -0,0 +1,10 @@
+---
+- name: device
+ fields:
+ - name: serial_number
+ level: core
+ type: keyword
+ short: Serial Number of the device
+ description: >
+ The unique serial number serves as a distinct identifier for each device, aiding in inventory management and device authentication.
+ example: DJGAQS4CW5

rfcs/text/0044/hash.yml

@@ -0,0 +1,9 @@
+---
+- name: file
+ fields:
+ - name: cdhash
+ level: extended
+ type: keyword
+ short: The Code Directory (CD) hash of an executable.
+ description: Code directory hash, utilized to uniquely identify and authenticate the integrity of the executable code.
+ example: 3783b4052fd474dbe30676b45c329e7a6d44acd9

rfcs/text/0044/process.yml

@@ -0,0 +1,36 @@
+---
+- name: process
+ title: Process
+ group: 2
+ short: These fields contain information about a process.
+ description: >
+ These fields contain information about a process.
+
+ These fields can help you correlate metrics information with a process id/name
+ from a log message. The `process.pid` often stays in the metric itself and is
+ copied to the global field for correlation.
+ type: group
+ reusable:
+ top_level: true
+ expected:
+ - at: process
+ as: responsible
+ short_override: Information about the responsible process.
+
+- name: process
+ fields:
+ - name: platform_binary
+ level: extended
+ type: boolean
+ short: Indicates whether this process executable is a default platform binary shipped with the operating system.
+ description: >
+ Binaries that are shipped by the operating system are defined as platform binaries, this value is then set to true. 
+ example: true
+
+ - name: endpoint_security_client
+ level: extended
+ type: boolean
+ short: Indicates whether this process executable is an Endpoint Security client.
+ description: >
+ Processes that have an endpoint security client must have the com.apple.endpointsecurity entitlement and the value is set to true in the message.
+ example: true

scripts/requirements-dev.txt

@@ -1,6 +1,6 @@
 # License: MIT
-autopep8==1.6.0
+autopep8==2.3.1
 # License: BSD
-mock==4.0.3
+mock==5.1.0
 # License: GPLv3
-yamllint==1.26.3
+yamllint==1.35.1

scripts/requirements.txt

@@ -1,7 +1,7 @@
 pip
 # License: MIT
-PyYAML==6.0.1
+PyYAML==6.0.2
 # License: BSD
-gitpython==3.1.41
+gitpython==3.1.43
 # License: BSD
 Jinja2==3.1.4

scripts/schema/subset_filter.py

@@ -121,7 +121,7 @@ def remove_docs_only_entries(paths: List[str], fields: Dict[str, FieldEntry]) ->
  split_path = path.split('.')
  field_set = split_path[0]
  field = split_path[1]
- del(fields[field_set]['fields'][field])
+ del (fields[field_set]['fields'][field])
  return fields
 
 

scripts/tests/test_asciidoc_fields.py

@@ -186,7 +186,7 @@ def dummy_nested_event_fieldset(self):
 
  def test_validate_sort_fieldset(self):
  sorted_foo_fields = asciidoc_fields.sort_fields(self.foo_fieldset)
- #import pdb;pdb.set_trace()
+ # import pdb;pdb.set_trace()
  self.assertIsInstance(sorted_foo_fields, list)
 
  # `allowed_value_names` always present

scripts/tests/test_ecs_helpers.py

@@ -144,7 +144,7 @@ def test_glob_yaml_files(self):
  self.assertEqual(ecs_helpers.glob_yaml_files('non_existent_wildcard.*'), [])
  self.assertEqual(ecs_helpers.glob_yaml_files('schemas/base.yml'), ['schemas/base.yml'])
  self.assertEqual(ecs_helpers.glob_yaml_files(['schemas/base.yml']), ['schemas/base.yml'])
- # convert to set as element order is not being tested
+ #  convert to set as element order is not being tested
  self.assertEqual(set(ecs_helpers.glob_yaml_files(
  ['schemas/base.yml', 'schemas/log.yml'])), {'schemas/base.yml', 'schemas/log.yml'})
  self.assertTrue(set(ecs_helpers.glob_yaml_files('schemas/b*.yml')).intersection({'schemas/base.yml'}) != set())