[RFC] 0017 Deprecation of log.original Stage 3 Proposal

rfcs/text/0017-remove-log-original.md

@@ -1,7 +1,7 @@
 # 0017: Remove log.original
 
-- Stage: **2 (candidate)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
-- Date: **2021-04-28** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->
+- Stage: **3 (finished)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html -->
+- Date: **2021-06-23** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. -->
 
 This RFC supersedes issue [#841](https:/elastic/ecs/issues/841) which implies breaking changes therefore the RFC Process is indicated.
 
@@ -45,10 +45,13 @@ Any or all incoming log or event messages when the original value of the event n
  dst=2.1.2.2spt=1232"}
 
 {"event.original": "Sep 19 08:26:10 localhost My log"}
+
+{
+ "event": {
+ "original": "{\"Node\":\"compute01-hc-xx-local-abcdefgh-com\",\"CheckID\":\"_nomad-check-2724899cc463354bb6336c67421b73f36fb966e2\",\"Name\":\"service: \\\"hello-world\\\" check\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"HTTP GET http://10.100.8.99:8080/: 200 OK Output: Hello world! Your request address: 10.100.8.34:37230\\n\\nRuntime env:\\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\\nHOSTNAME=7c9b09a0258c\\nNOMAD_ALLOC_DIR=/alloc\\nNOMAD_ALLOC_ID=4c62244a-6924-4cb6-60d3-a4c574c58085\\nNOMAD_ALLOC_INDEX=0\\nNOMAD_ALLOC_NAME=hello-world.servers[0]\\nNOMAD_CPU_LIMIT=50\\nNOMAD_DC=xx\\nNOMAD_GROUP_NAME=servers\\nNOMAD_JOB_ID=hello-world\\nNOMAD_JOB_NAME=hello-world\\nNOMAD_MEMORY_LIMIT=32\\nNOMAD_NAMESPACE=default\\nNOMAD_REGION=global\\nNOMAD_SECRETS_DIR=/secrets\\nNOMAD_TASK_DIR=/local\\nNOMAD_TASK_NAME=hello-world\\nHOME=/\\n\",\"ServiceID\":\"_nomad-task-4c62244a-6924-4cb6-60d3-a4c574c58085-group-servers-hello-world-8080\",\"ServiceName\":\"hello-world\",\"ServiceTags\":[\"http\",\"traefik.enable=true\",\"traefik.http.routers.hello-world.entrypoints=https\",\"traefik.http.routers.hello-world.tls.certresolver=le\",\"traefik.http.routers.hello-world.tls.domains[0].main=svc.abcdefgh.com\",\"traefik.http.routers.hello-world.tls.domains[0].sans=*.svc.abcdefgh.com\"],\"Type\":\"http\",\"Definition\":{},\"CreateIndex\":4281349\"ModifyIndex\":5749556}"
+ }
+}
 ```
-<!--
-Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2.
--->
 
 ## Scope of impact
 
@@ -58,9 +61,13 @@ The removal of `log.original` will be considered a breaking change since the fie
 
 - The [`Beats default fields inclusion list`](https:/elastic/ecs/blob/master/scripts/generators/beats_default_fields_allowlist.yml) list should be updated by removing `log.original` if/when Deprecation progresses to Removal
 
-- The logs UI `message` column currently displays `log.original` in the absence of a `message` field. This should be updated to use `event.original` as the substitute field. See [builtin_rules](https:/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.ts) and [associated test](https:/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.test.ts).
+- The logs UI `message` column currently displays `log.original` in the absence of a `message` field. It should be updated to use `event.original` as the substitute field. See [builtin_rules](https:/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.ts) and [associated test](https:/elastic/kibana/blob/master/x-pack/plugins/infra/server/services/log_entries/message/builtin_rules/generic.test.ts). Kibana team is aware and has a [PR awaiting review](https:/elastic/kibana/pull/102236) which addresses this.
+
+- The breakdown of usage of this field in Kibana source shows:
+ - 88 usages in tests
+ - 22 usages outside of tests, e.g. `js`, `ts`, `map` files
 
-- References in the [RAC Rule Registry](https:/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts) will need to be removed - these have `required: false` so hopefully non-breaking change.
+- For example, references in the [RAC Rule Registry](https:/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts) will need to be removed - these have `required: false` so hopefully non-breaking change.
 
 - Multiple tests in Kibana will need to be updated see e.g. [Function Test APM Mapping](https:/elastic/kibana/blob/master/x-pack/test/functional/es_archives/monitoring/setup/collection/detect_apm/mappings.json)
 
@@ -72,9 +79,13 @@ The removal of `log.original` will be considered a breaking change since the fie
 
 As a breaking change, this would require timely communication to the Elastic Community.
 
-<!--
-Stage 3: Document resolutions for all existing concerns. Any new concerns should be documented along with their resolution. The goal here is to eliminate risk of churn and instability by ensuring all concerns have been addressed.
--->
+With respect to internal stakeholders, the following actons have been taken:
+
+- Reached out the Beats + Integrations teams: Filebeat has already taken action, thanks @P1llus
+
+- Reached out to Logs UI team: `logs.original` has now been superceed by `event.original` as fallback when `message`not available, thanks @weltenwort
+
+- Reached out to RAC: the reference [RAC Rule Registry](https:/elastic/kibana/blob/master/x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts) has been acknowledged, thanks @jasonrhodes
 
 ## People
 
@@ -86,11 +97,16 @@ The following are the people that consulted on the contents of this RFC.
 * @jasonrhodes | Logging UI & RAC
 * @MikePaquette | RAC
 
-
 ## References
 
-* [#841](https:/elastic/ecs/issues/841)
-* [#777](https:/elastic/integrations/issues/777)
+### Issues
+* [ecs #841](https:/elastic/ecs/issues/841)
+* [integrations #777](https:/elastic/integrations/issues/777)
+* [integrations #994](https:/elastic/integrations/issues/994)
+* [logs UI #101307](https:/elastic/kibana/issues/101307)
+
+### Stack PRs
+* [kibana PR #102236](https:/elastic/kibana/pull/102236)
 
 ### RFC Pull Requests
 
@@ -99,3 +115,5 @@ The following are the people that consulted on the contents of this RFC.
 * Stage 0: https:/elastic/ecs/pull/1298
 * Stage 1: https:/elastic/ecs/pull/1314
 * Stage 2: https:/elastic/ecs/pull/1347
+* Stage 3: https:/elastic/ecs/pull/1465
+* Implementation: https:/elastic/ecs/pull/1469