Merge branch 'main' into rfc/0010/stage-2

.backportrc.json

@@ -2,6 +2,7 @@
  "upstream": "elastic/ecs",
  "branches": [
  { "name": "master", "checked": true },
+ "8.0",
  "1.12",
  "1.11",
  "1.10",
@@ -19,7 +20,7 @@
  "targetPRLabels": ["backport"],
  "prFilter": "label:needs_backport",
  "branchLabelMapping": {
- "^8.0.0$": "master",
+ "^8.1.0$": "master",
  "^(\\d+).(\\d+).\\d+$": "$1.$2"
  }
 }

.github/ISSUE_TEMPLATE/schema-changes-additions.md

@@ -7,7 +7,7 @@ labels: "enhancement"
 <!--
 Please first search existing issues for the changes you are requesting; it may already exist as an open issue.
 
-Substantial schema changes or additions should follow the RFC process: https:/elastic/ecs/blob/master/rfcs/README.md
+Substantial schema changes or additions should follow the RFC process: https:/elastic/ecs/blob/main/rfcs/README.md
 
 Please fill in the following sections describing your proposed changes: -->
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -6,9 +6,9 @@ our submission, but they are here to help bring them to your attention.
 -->
 
 - Have you signed the [contributor license agreement](https://www.elastic.co/contributor-agreement)?
-- Have you followed the [contributor guidelines](https:/elastic/ecs/blob/master/CONTRIBUTING.md)?
-- For proposing substantial changes or additions to the schema, have you reviewed the [RFC process](https:/elastic/ecs/blob/master/rfcs/README.md)?
+- Have you followed the [contributor guidelines](https:/elastic/ecs/blob/main/CONTRIBUTING.md)?
+- For proposing substantial changes or additions to the schema, have you reviewed the [RFC process](https:/elastic/ecs/blob/main/rfcs/README.md)?
 - If submitting code/script changes, have you verified all tests pass locally using `make test`?
 - If submitting schema/fields updates, have you generated new artifacts by running `make` and committed those changes?
-- Is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
-- Have you added an entry to the [CHANGELOG.next.md](https:/elastic/ecs/blob/master/CHANGELOG.next.md)?
+- Is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
+- Have you added an entry to the [CHANGELOG.next.md](https:/elastic/ecs/blob/main/CHANGELOG.next.md)?

CHANGELOG.md

@@ -3,6 +3,57 @@
 # CHANGELOG
 All notable changes to this project will be documented in this file based on the [Keep a Changelog](http://keepachangelog.com/) Standard. This project adheres to [Semantic Versioning](http://semver.org/).
 
+## [1.12.1](https:/elastic/ecs/compare/v1.12.0...v1.12.1)
+
+### Schema Changes
+
+#### Bugfixes
+
+* Updating `x509` order to correct nesting. ##1621
+
+## [1.12.0](https:/elastic/ecs/compare/v1.11.0...v1.12.0)
+
+### Schema Changes
+
+#### Bugfixes
+
+* Updating `hash` order to correct nesting. #1603
+* Removing incorrect `hash` reuses. #1604
+* Updating `pe` order to correct nesting. #1605
+* Removing incorrect `pe` reuses. #1606
+* Correcting `enrichments` to an `array` type. #1608
+
+#### Added
+
+* Added `file.fork_name` field. #1288
+* Added `service.address` field. #1537
+* Added `service.environment` as a beta field. #1541
+* Added `process.end` field. #1544
+* Added container metric fields into experimental schema. #1546
+* Add `code_signature.digest_algorithm` and `code_signature.timestamp` fields. #1557
+* Add `email.*` field set in the experimental fields. #1569
+
+#### Improvements
+
+* Beta migration on some `keyword` fields to `wildcard`. #1517
+* Promote `threat.software.*` and `threat.group.*` fields to GA. #1540
+* Update `user.name` and `user.id` examples for clarity. #1566
+* Beta migration of `text` and `.text` multi-fields to `match_only_text`. #1532, #1571
+
+### Tooling and Artifact Changes
+
+#### Added
+
+* Support ES 6.x type fallback for `match_only_text` field types. #1528
+
+#### Bugfixes
+
+* Prevent failure if no files need to be deleted `find | xargs rm`. #1588
+
+#### Improvements
+
+* Document field type family interoperability in FAQ. #1591
+
 ## [1.11.0](https:/elastic/ecs/compare/v1.10.0...v1.11.0)
 
 ### Schema Changes

CHANGELOG.next.md

@@ -12,67 +12,68 @@ Thanks, you're awesome :-) -->
 
 #### Breaking changes
 
-* Remove `host.user.*` field reuse. #1439
-* Remove deprecation notice on `http.request.method`. #1443
-* Migrate `log.origin.file.line` from `integer` to `long`. #1533
-* Remove `log.original` field. #1580
-
 #### Bugfixes
 
 #### Added
 
 #### Improvements
 
-* Wildcard type field migration GA. #1582
-* `match_only_text` type field migration GA. #1584
-* Threat indicator fields GA from RFC 0008. #1586
-
 #### Deprecated
 
-### Tooling and Artifact Changes
+#### Removed
 
-#### Breaking Changes
+- Removing `process.target.*` reuses from experimental schema. #1666
 
-* Removing deprecated --oss from generator #1404
-* Removing use-cases directory #1405
-* Remove Go code generator. #1567
+### Tooling and Artifact Changes
+
+#### Breaking changes
 
 #### Bugfixes
 
+* Add `object` as fallback for `flattened` type. #1653
+
 #### Added
 
 #### Improvements
 
-* Remove remaining Go deps after removing Go code generator. #1585
+* Update refs from master to main in USAGE.md etc #1658
 
 #### Deprecated
 
-## 1.12.0 (Feature Freeze)
+## 8.0.0 (Feature Freeze)
 
 ### Schema Changes
 
+#### Breaking changes
+
+* Remove `host.user.*` field reuse. #1439
+* Remove deprecation notice on `http.request.method`. #1443
+* Migrate `log.origin.file.line` from `integer` to `long`. #1533
+* Remove `log.original` field. #1580
+* Remove `process.ppid` field. #1596
+
 #### Added
 
-* Added `file.fork_name` field. #1288
-* Added `service.address` field. #1537
-* Added `service.environment` as a beta field. #1541
-* Added `process.end` field. #1544
-* Added container metric fields into experimental schema. #1546
-* Add `code_signature.digest_algorithm` and `code_signature.timestamp` fields. #1557
-* Add `email.*` field set in the experimental fields. #1569
+* Added `faas.*` field set as beta. #1628
 
 #### Improvements
 
-* Beta migration on some `keyword` fields to `wildcard`. #1517
-* Promote `threat.software.*` and `threat.group.*` fields to GA. #1540
-* Update `user.name` and `user.id` examples for clarity. #1566
-* Beta migration of `text` and `.text` multi-fields to `match_only_text`. #1532, #1571
+* Wildcard type field migration GA. #1582
+* `match_only_text` type field migration GA. #1584
+* Threat indicator fields GA from RFC 0008. #1586
 
 ### Tooling and Artifact Changes
 
-#### Added
+#### Breaking Changes
 
-* Support ES 6.x type fallback for `match_only_text` field types. #1528
+* Removing deprecated --oss from generator #1404
+* Removing use-cases directory #1405
+* Remove Go code generator. #1567
+
+#### Improvements
+
+* Remove remaining Go deps after removing Go code generator. #1585
+* Add explicit `default_field: true` for Beats artifacts. #1633
 
 <!-- All empty sections:
 

CONTRIBUTING.md

@@ -101,7 +101,7 @@ Please follow these guidelines when submitting Issues:
 
 ECS follows this branching strategy:
 
-* The `master` is the next major version. It is where all new contributions are first merged. This includes new features and bug fixes, and it may also include breaking changes.
+* The `main` is the next major version. It is where all new contributions are first merged. This includes new features and bug fixes, and it may also include breaking changes.
 * The `<major>.x` is the next minor version and gets backports of most non-breaking features and fixes.
 * The `<major>.<minor>` is the next release of a minor version, including patch releases.
 
@@ -116,7 +116,7 @@ Breaking changes intended for the next major version should be included undernea
 
 ### Backports
 
-ECS maintains multiple release branches in the repo. The `master` branch is where all new contributions should be submitted, and features and bug fixes will be backported into other branches when appropriate. Any backporting needs will be handled by the ECS team.
+ECS maintains multiple release branches in the repo. The `main` branch is where all new contributions should be submitted, and features and bug fixes will be backported into other branches when appropriate. Any backporting needs will be handled by the ECS team.
 
 #### Tooling
 

Makefile

@@ -24,6 +24,15 @@ check: generate experimental test fmt misspell makelint
  git update-index --refresh
  git diff-index --exit-code HEAD --
 
+# Check for license headers
+.PHONY: check_license_headers
+check_license_headers:
+ @echo "Files missing license headers:\n"
+ @find . -type f \( -path './scripts/*' -o -path './schemas/*' \) \
+ \( -name '*.py' -o -name '*.yml' \) \
+ -print0 | xargs -0 -n1 grep -L "Licensed to Elasticsearch B.V." \
+ || exit 0
+
 # Clean deletes all temporary and generated content.
 .PHONY: clean
 clean:

NOTICE.txt

@@ -1,5 +1,5 @@
 Elastic Common Schema
-Copyright 2018 Elasticsearch B.V.
+Copyright 2018-2021 Elasticsearch B.V.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

README.md

@@ -27,10 +27,28 @@ You can learn more in [generated/README.md](generated)
 
 ## Releases of ECS
 
-The master branch of this repository should never be considered an
+The main branch of this repository should never be considered an
 official release of ECS. You can browse official releases of ECS
 [here](https:/elastic/ecs/releases).
 
 The ECS team publishes improvements to the schema by following
 [Semantic Versioning](https://semver.org/).
 Generally major ECS releases are planned to be aligned with major Elastic Stack releases.
+
+## License
+
+This software is licensed under the Apache License, version 2 ("ALv2"), quoted below.
+
+Copyright 2018-2021 Elasticsearch <https://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy of
+the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations under
+the License.

USAGE.md

@@ -135,7 +135,7 @@ Running generator. ECS version 1.5.0
 **Points to note on the defaults**:
 
 * Artifacts are created in the [`generated`](generated) directory and the entire schema is included
-* Documentation updates will be written to the appropriate file under the `docs` directory. More specifics on generated doc files is covered in the [contributor's file](https:/elastic/ecs/blob/master/CONTRIBUTING.md#generated-documentation-files)
+* Documentation updates will be written to the appropriate file under the `docs` directory. More specifics on generated doc files is covered in the [contributor's file](https:/elastic/ecs/blob/main/CONTRIBUTING.md#generated-documentation-files)
 * Each run of the script will rewrite the entirety of the `generated` directory
 * The script will need to be executed from the top-level of the ECS repo
 * The `version` displayed when running `generator.py` is based on the current value of the [version](version) file in the top-level of the repo
@@ -164,7 +164,7 @@ Use the `--include` flag to generate ECS artifacts based on the current ECS sche
 $ python scripts/generator.py --include ../myproject/ecs/custom-fields/
 ```
 
-The `--include` flag expects a directory of schema YAML files using the same [file format](https:/elastic/ecs/tree/master/schemas#fields-supported-in-schemasyml) as the ECS schema files. This is useful for maintaining custom field definitions that are _outside_ of the ECS schema, but allows for merging the custom fields with the official ECS fields for your deployment.
+The `--include` flag expects a directory of schema YAML files using the same [file format](https:/elastic/ecs/tree/main/schemas#fields-supported-in-schemasyml) as the ECS schema files. This is useful for maintaining custom field definitions that are _outside_ of the ECS schema, but allows for merging the custom fields with the official ECS fields for your deployment.
 
 For example, if we defined the following schema definition in a file named `myproject/ecs/custom-fields/widget.yml`:
 
@@ -234,14 +234,14 @@ Include can be used together with the `--ref` flag to merge custom fields into a
 
 #### Exclude
 
-Use the `--exclude` flag to generate ephemeral ECS artifacts based on the current ECS schema field definitions minus fields considered for removal, e.g. to assess impact of removing these. Warning! This is not the recommended route to remove a field permanently as it is not intentended to be invoked during the build process. Definitive field removal should be implemented using a custom [Subset](#subset) or via the [RFC process](https:/elastic/ecs/tree/master/rfcs/README.md). Example:
+Use the `--exclude` flag to generate ephemeral ECS artifacts based on the current ECS schema field definitions minus fields considered for removal, e.g. to assess impact of removing these. Warning! This is not the recommended route to remove a field permanently as it is not intended to be invoked during the build process. Definitive field removal should be implemented using a custom [Subset](#subset) or via the [RFC process](https:/elastic/ecs/tree/main/rfcs/README.md). Example:
 
 ```
 $ python scripts/generator.py --exclude=../my-project/my-exclude-file.yml
 $ python scripts/generator.py --exclude="../my-project/schemas/a*.yml"
 ```
 
-The `--exclude` flag expects a path to one or more YAML files using the same [file format](https:/elastic/ecs/tree/master/schemas#fields-supported-in-schemasyml) as the ECS schema files. You can also use a subset, provided that relevant `name` and `fields` fields are preserved.
+The `--exclude` flag expects a path to one or more YAML files using the same [file format](https:/elastic/ecs/tree/main/schemas#fields-supported-in-schemasyml) as the ECS schema files. You can also use a subset, provided that relevant `name` and `fields` fields are preserved.
 
 ```
 ---

docs/converting.asciidoc

@@ -17,8 +17,8 @@ Before you start a conversion, be sure that you understand the basics below.
 Make sure you understand the distinction between Core and Extended fields,
 as explained in the <<ecs-guidelines>>.
 
-Core and Extended fields are documented in the <<ecs-field-reference>> or, for 
-a single page representation of all fields, please see the 
+Core and Extended fields are documented in the <<ecs-field-reference>> or, for
+a single page representation of all fields, please see the
 {ecs_github_repo_link}/generated/csv/fields.csv[generated CSV of fields].
 
 [float]

docs/faq.asciidoc

@@ -96,3 +96,20 @@ the ECS data itself, this is not an issue because all fields are predefined.
 
 As long as there are no conflicts, underline notation and ECS dot notation can
 coexist in the same document.
+
+[float]
+[[type-interop]]
+==== What if I want to use a different data type from the same field type family?
+
+In Elasticsearch, field types are grouped by family. Types in the same family support
+the same search functionality but may have different space usage or performance
+characteristics. For example, both `keyword` and `wildcard` types are members of the
+`keyword` family, and `text` and `match_only_text` are members of the `text` family.
+
+The field types defined in ECS provide the best default experience for most users.
+However, a different type from the same family can replace the default defined in ECS
+if required for a specific use cases. Users should understand any potential performance
+or storage differences before changing from a default field type.
+
+The Elasticsearch {ref}/mapping-types.html[mapping types] section has more information about type
+families.