From d8724427d6ff6143d35cf8597f615e45bcd6b9ce Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Tue, 15 Mar 2022 13:34:49 +0100 Subject: [PATCH 01/33] OTEP: Support Elastic Common Schema in OpenTelemetry --- ...-elastic-common-schema-in-opentelemetry.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 text/0000-support-elastic-common-schema-in-opentelemetry.md diff --git a/text/0000-support-elastic-common-schema-in-opentelemetry.md b/text/0000-support-elastic-common-schema-in-opentelemetry.md new file mode 100644 index 000000000..2286b3cfc --- /dev/null +++ b/text/0000-support-elastic-common-schema-in-opentelemetry.md @@ -0,0 +1,253 @@ +# Support Elastic Common Schema in OpenTelemetry + +Collaborators: Alolita Sharma (OTEL GC, AWS), Cyrille Le Clerc (Elastic), Daniel Khan (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) + + +## Introduction + +This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. + + +## Motivation + +Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals. + +Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTEL Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition. + +In addition to the use case of structured logs, the maturity of ECS for SIEM (Security Information and Event Management) is a great opportunity for OpenTelemetry to expand its scope to the security use cases. + +Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type’ to identify event types. + +We’d like to see different categories of structured logs being well-supported in the [OTEL Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). + + +## Customer Motivation + +Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTEL Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor generated along with open source logs. + +Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL compatible observability products and services. + +OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)). + +The semantic conventions of a log are a challenge. What is a specific component defined in a log and how does it relate to other logs which have the same semantic component defined differently. ECS has already done some heavy-lifting on defining a unified set of semantic conventions which can be adopted in OTEL. + +OpenTelemetry has the potential to grow exponentially if the data from these other services can be correlated with instrumented code and components. In order to do this, industry stakeholders should leverage a common and standard logging data model which allows for the mapping of these different data types. The OpenTelemetry data protocol can provide this interoperable open standard. This unlocks countless use cases, and ensures that OpenTelemetry can work with other technologies which are not OpenTelemetry compliant. + + +## Background + + +### What is ECS? + +The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic’s user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) + +The coverage of ECS is very broad including in depth support for logs, security, and network events such as "[logs.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-log.html)" , "[geo.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html)", "[tls.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html)", "[dns.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-dns.html)", or "[vulnerability.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html)". + +ECS has the following guiding principles: + +* ECS favors human readability in order to enable broader adoption, +* ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...), + * ECS doesn’t differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes), +* ECS groups fields in namespaces in order to: + * Offer consistency and readability, + * Enable reusability of namespaces in different contexts, + * Example the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces + * Enable extensibility adding fields to namespaces and adding new namespaces, + * Prevent field name conflicts +* ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It’s much broader than simple logging use cases. + + +### Example of a log message structured with ECS: NGINX access logs + +Example of an NGinx Access Log entry structured with ECS + + +```json +{ + "@timestamp":"2020-03-25T09:51:23.000Z", + "client":{ + "ip":"10.42.42.42" + }, + "http":{ + "request":{ + "referrer":"-", + "method":"GET" + }, + "response":{ + "status_code":200, + "body":{ + "bytes":2571 + } + }, + "version":"1.1" + }, + "url":{ + "path":"/blog" + }, + "user_agent":{ + "original":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36", + "os":{ + "name":"Mac OS X", + "version":"10.14.0", + "full":"Mac OS X 10.14.0" + }, + "name":"Chrome", + "device":{ + "name":"Other" + }, + "version":"70.0.3538.102" + }, + "log":{ + "file":{ + "path":"/var/log/nginx/access.log" + }, + "offset":33800 + }, + "host": { + "hostname": "cyrille-laptop.home", + "os": { + "build": "19D76", + "kernel": "19.3.0", + "name": "Mac OS X", + "family": "darwin", + "version": "10.15.3", + "platform": "darwin" + }, + "name": "cyrille-laptop.home", + "id": "04A12D9F-C409-5352-B238-99EA58CAC285", + "architecture": "x86_64" + } + +} +``` + + + +## Comparison between OpenTelemetry Semantic Conventions for logs and ECS + + +## Principles + + + +| Description | [OTel Logs and Event Record](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition) | [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) | +|-------------|-------------|--------| +| Metadata shared by all the Log Messages / Spans of an application instance | Resource Attributes | ECS fields | +| Metadata specific to each Log Message / Span | Attributes | ECS Fields | +| Message of log events | Body | [message field](https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-message) | +| Naming convention | Dotted names | Dotted names | +| Reusability of namespaces | Namespaces are intended to be composed | Namespaces are intended to be composed | +| Extensibility | Attributes can be extended by either adding a user defined field to an existing namespaces or introducing new namespaces. | Extra attributes can be added in each namespace and users can create their own namespaces | + +## Data Types + +| Category | OTel Logs and Event Record (all or a subset of GRPC data types) | ECS Data Types | +|---|---|---| +| Text | string | text, match_only_text, keyword constant_keyword, wildcard | +| Dates | uint64 nanoseconds since Unix epoch | date, date_nanos | +| Numbers | number | long, double, scaled_float, boolean… | +| Objects | Uint32, uint64… | object (JSON object), flattened (An entire JSON object as a single field value) | +| Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in Otel).
Note that Otel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | +| Binary data | Byte sequence | binary | + + + + +## Known Differences + +Some differences exist on fields that are both defined in OpenTelemetry Semantic Conventions and in ECS. In this case, it would make sense for overlapping ECS fields to not be integrated in the new specification. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OTel Logs and Event Record + Elastic Common Schema (ECS) + Description +
Timestamp (uint64 nanoseconds since Unix epoch) + @timestamp (date) + +
TraceId (byte sequence), SpanId (byte sequence) + trace.id (keyword), span.id (keyword) + +
N/A + Transaction.id (keyword) + +
SeverityText (string) + log.syslog.severity.name (keyword), log.level (keyword) + +
SeverityNumber (number) + log.syslog.severity.code + +
Body (any) + message (match_only_text) + +
process.cpu.load (not specified but collected by Otel Collector) +

+process.cpu.time (async counter) +

+system.cpu.utilization +

host.cpu.usage (scaled_float) with a slightly different measurement than what otel metrics measure + Note that most metrics have slightly different names and semantics between ECS and OpenTelemetry +
+ + + +# Alternatives / Discussion + + +## Prometheus Naming Conventions + +Prometheus is a de facto standard for observability metrics and OpenTelemetry already provides full interoperability with the Prometheus ecosystem. + +It would be useful to get interoperability between metrics collected by an [official Prometheus exporter](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and the equivalent OpenTelemetry Collector receivers (e.g. Otel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver)) + +Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which don’t enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts. + +Prometheus conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don’t specify unified metric names. + + +# Other areas that need to be addressed by OTEL (the project) + +Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)) From db20eef0a3c3fd78806b2aa81e6cc3aa92e725fd Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Tue, 15 Mar 2022 14:11:53 +0100 Subject: [PATCH 02/33] Fix typos --- ...-elastic-common-schema-in-opentelemetry.md | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/text/0000-support-elastic-common-schema-in-opentelemetry.md b/text/0000-support-elastic-common-schema-in-opentelemetry.md index 2286b3cfc..56082fe7d 100644 --- a/text/0000-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0000-support-elastic-common-schema-in-opentelemetry.md @@ -16,9 +16,9 @@ Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTele In addition to the use case of structured logs, the maturity of ECS for SIEM (Security Information and Event Management) is a great opportunity for OpenTelemetry to expand its scope to the security use cases. -Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type’ to identify event types. +Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type' to identify event types. -We’d like to see different categories of structured logs being well-supported in the [OTEL Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). +We'd like to see different categories of structured logs being well-supported in the [OTEL Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). ## Customer Motivation @@ -39,7 +39,7 @@ OpenTelemetry has the potential to grow exponentially if the data from these oth ### What is ECS? -The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic’s user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) +The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic's user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) The coverage of ECS is very broad including in depth support for logs, security, and network events such as "[logs.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-log.html)" , "[geo.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html)", "[tls.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html)", "[dns.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-dns.html)", or "[vulnerability.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html)". @@ -47,19 +47,19 @@ ECS has the following guiding principles: * ECS favors human readability in order to enable broader adoption, * ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...), - * ECS doesn’t differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes), + * ECS doesn't differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes), * ECS groups fields in namespaces in order to: * Offer consistency and readability, * Enable reusability of namespaces in different contexts, - * Example the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces + * For example, the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces * Enable extensibility adding fields to namespaces and adding new namespaces, * Prevent field name conflicts -* ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It’s much broader than simple logging use cases. +* ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It's much broader than simple logging use cases. ### Example of a log message structured with ECS: NGINX access logs -Example of an NGinx Access Log entry structured with ECS +Example of a Nginx Access Log entry structured with ECS ```json @@ -128,8 +128,6 @@ Example of an NGinx Access Log entry structured with ECS ## Principles - - | Description | [OTel Logs and Event Record](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition) | [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) | |-------------|-------------|--------| | Metadata shared by all the Log Messages / Spans of an application instance | Resource Attributes | ECS fields | @@ -146,7 +144,7 @@ Example of an NGinx Access Log entry structured with ECS | Text | string | text, match_only_text, keyword constant_keyword, wildcard | | Dates | uint64 nanoseconds since Unix epoch | date, date_nanos | | Numbers | number | long, double, scaled_float, boolean… | -| Objects | Uint32, uint64… | object (JSON object), flattened (An entire JSON object as a single field value) | +| Objects | uint32, uint64… | object (JSON object), flattened (An entire JSON object as a single field value) | | Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in Otel).
Note that Otel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | | Binary data | Byte sequence | binary | @@ -241,13 +239,13 @@ As the markdown code of the tables is hard to read and maintain with very long l Prometheus is a de facto standard for observability metrics and OpenTelemetry already provides full interoperability with the Prometheus ecosystem. -It would be useful to get interoperability between metrics collected by an [official Prometheus exporter](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and the equivalent OpenTelemetry Collector receivers (e.g. Otel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver)) +It would be useful to get interoperability between metrics collected by [official Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and their equivalent OpenTelemetry Collector receivers (e.g. Otel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver)). -Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which don’t enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts. +Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which doesn't enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts. -Prometheus conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don’t specify unified metric names. +Prometheus' conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don't specify unified metric names. # Other areas that need to be addressed by OTEL (the project) -Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)) +Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). From 00a69967d8275b11338ef219a953ac5554dc0e7b Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 17 Mar 2022 00:20:38 +0100 Subject: [PATCH 03/33] Clarify the proposed process for this OTEP + example of the practical benefits of this OTEP --- ...-elastic-common-schema-in-opentelemetry.md | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/text/0000-support-elastic-common-schema-in-opentelemetry.md b/text/0000-support-elastic-common-schema-in-opentelemetry.md index 56082fe7d..0bf279af6 100644 --- a/text/0000-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0000-support-elastic-common-schema-in-opentelemetry.md @@ -5,8 +5,16 @@ Collaborators: Alolita Sharma (OTEL GC, AWS), Cyrille Le Clerc (Elastic), Daniel ## Introduction -This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. +This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into Otel Semantic Conventions. +## Proposed process to merge ECS in OpenTelemetry Semantic Conventions + +We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions: +1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented merging ECS fields in OpenTelemetry Semantic Conventions, +2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of Otel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), +3. Actual merge of ECS fields in Otel Semantic Conventions. + +Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we prefered to first validate the principle and, once consensus is reached, actually define the mapping. ## Motivation @@ -36,7 +44,6 @@ OpenTelemetry has the potential to grow exponentially if the data from these oth ## Background - ### What is ECS? The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic's user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) @@ -125,7 +132,6 @@ Example of a Nginx Access Log entry structured with ECS ## Comparison between OpenTelemetry Semantic Conventions for logs and ECS - ## Principles | Description | [OTel Logs and Event Record](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition) | [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) | @@ -148,9 +154,6 @@ Example of a Nginx Access Log entry structured with ECS | Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in Otel).
Note that Otel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | | Binary data | Byte sequence | binary | - - - ## Known Differences Some differences exist on fields that are both defined in OpenTelemetry Semantic Conventions and in ECS. In this case, it would make sense for overlapping ECS fields to not be integrated in the new specification. @@ -230,12 +233,17 @@ As the markdown code of the tables is hard to read and maintain with very long l +## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS + +### Example an OpenTelemetry Collector Receiver to collect the access logs of a web server +The author of the "Otel Collector Access logs file receiver for web server XXX" would find in the Otel Semantic Convention specifications all the guidance to map the fields of the web server logs, not only the attributes that the Otel Semantic Conventions has specified today for [HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). +This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions and enabling richer correlations. -# Alternatives / Discussion +## Alternatives / Discussion -## Prometheus Naming Conventions +### Prometheus Naming Conventions Prometheus is a de facto standard for observability metrics and OpenTelemetry already provides full interoperability with the Prometheus ecosystem. @@ -246,6 +254,6 @@ Note that one of the challenges with Prometheus metrics naming conventions is th Prometheus' conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don't specify unified metric names. -# Other areas that need to be addressed by OTEL (the project) +## Other areas that need to be addressed by OTEL (the project) Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). From f0fec03b0b57546da6bb5529a1410a68d6a0fb70 Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 31 Mar 2022 16:32:00 +0200 Subject: [PATCH 04/33] Apply suggestions from code review Co-authored-by: Armin Ruech --- ...-elastic-common-schema-in-opentelemetry.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/text/0000-support-elastic-common-schema-in-opentelemetry.md b/text/0000-support-elastic-common-schema-in-opentelemetry.md index 0bf279af6..46b677b6e 100644 --- a/text/0000-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0000-support-elastic-common-schema-in-opentelemetry.md @@ -5,12 +5,12 @@ Collaborators: Alolita Sharma (OTEL GC, AWS), Cyrille Le Clerc (Elastic), Daniel ## Introduction -This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into Otel Semantic Conventions. +This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into OTel Semantic Conventions. ## Proposed process to merge ECS in OpenTelemetry Semantic Conventions We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions: -1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented merging ECS fields in OpenTelemetry Semantic Conventions, +1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented by merging ECS fields in OpenTelemetry Semantic Conventions, 2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of Otel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), 3. Actual merge of ECS fields in Otel Semantic Conventions. @@ -18,7 +18,7 @@ Note that we didn't propose in this OTEP the mapping of all ECS fields because t ## Motivation -Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals. +Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor-created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals. Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTEL Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition. @@ -31,9 +31,9 @@ We'd like to see different categories of structured logs being well-supported in ## Customer Motivation -Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTEL Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor generated along with open source logs. +Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTEL Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated along with open source logs. -Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL compatible observability products and services. +Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL-compatible observability products and services. OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)). @@ -52,14 +52,14 @@ The coverage of ECS is very broad including in depth support for logs, security, ECS has the following guiding principles: -* ECS favors human readability in order to enable broader adoption, +* ECS favors human readability in order to enable broader adoption as many fields can be understood without having to read up their meaning in the reference, * ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...), - * ECS doesn't differentiate the metadata fields that are specific to each event of the event source and the metadata that are shared by all the events of the source the same way OTEL differentiates Resource Attributes and Log/Span Attributes), + * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by all the events of the source in the way OTEL does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, * ECS groups fields in namespaces in order to: * Offer consistency and readability, * Enable reusability of namespaces in different contexts, * For example, the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces - * Enable extensibility adding fields to namespaces and adding new namespaces, + * Enable extensibility by adding fields to namespaces and adding new namespaces, * Prevent field name conflicts * ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It's much broader than simple logging use cases. @@ -136,8 +136,8 @@ Example of a Nginx Access Log entry structured with ECS | Description | [OTel Logs and Event Record](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#log-and-event-record-definition) | [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) | |-------------|-------------|--------| -| Metadata shared by all the Log Messages / Spans of an application instance | Resource Attributes | ECS fields | -| Metadata specific to each Log Message / Span | Attributes | ECS Fields | +| Metadata shared by all the Log Messages / Spans / Metrics of an application instance | Resource Attributes | ECS fields | +| Metadata specific to each Log Message / Span / Metric data point | Attributes | ECS Fields | | Message of log events | Body | [message field](https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-message) | | Naming convention | Dotted names | Dotted names | | Reusability of namespaces | Namespaces are intended to be composed | Namespaces are intended to be composed | @@ -251,7 +251,7 @@ It would be useful to get interoperability between metrics collected by [officia Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which doesn't enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts. -Prometheus' conventions are restricted to the style of the name of the metrics (see [Metric and label naming | Prometheus](https://prometheus.io/docs/practices/naming/) ) but don't specify unified metric names. +Prometheus' conventions are restricted to the style of the name of the metrics (see [Prometheus Metric and label naming](https://prometheus.io/docs/practices/naming/)) but don't specify unified metric names. ## Other areas that need to be addressed by OTEL (the project) From db0d761ecae706f7d9f0866d3197d60b5bd52712 Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 31 Mar 2022 21:09:03 +0200 Subject: [PATCH 05/33] Rename OTEP file name --- ....md => 0199-support-elastic-common-schema-in-opentelemetry.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-support-elastic-common-schema-in-opentelemetry.md => 0199-support-elastic-common-schema-in-opentelemetry.md} (100%) diff --git a/text/0000-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md similarity index 100% rename from text/0000-support-elastic-common-schema-in-opentelemetry.md rename to text/0199-support-elastic-common-schema-in-opentelemetry.md From bd52ac5858d8fae5154d5415fc180a9088d7ac70 Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 14 Apr 2022 15:00:16 +0200 Subject: [PATCH 06/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Armin Ruech --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 46b677b6e..69c15d443 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -36,6 +36,8 @@ Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL-compatible observability products and services. OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)). +By expanding the OTel semantic conventions with further namespaces already defined in ECS, a broader coverage of such mappings from different sources can be defined and implemented in the OTel collector. +This, for example, includes logs from network appliances (mapping to the `network` and `interface` namespaces in ECS). The semantic conventions of a log are a challenge. What is a specific component defined in a log and how does it relate to other logs which have the same semantic component defined differently. ECS has already done some heavy-lifting on defining a unified set of semantic conventions which can be adopted in OTEL. From aa71030a8f9a43d878ac9002541437ea863e378e Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Fri, 2 Sep 2022 14:53:48 +0200 Subject: [PATCH 07/33] Update collaborators list --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 69c15d443..da965d327 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -1,6 +1,6 @@ # Support Elastic Common Schema in OpenTelemetry -Collaborators: Alolita Sharma (OTEL GC, AWS), Cyrille Le Clerc (Elastic), Daniel Khan (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) +Collaborators: Alolita Sharma (OTEL GC, Apple), Cyrille Le Clerc (Elastic), Daniel Khan (Sentry), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) ## Introduction From de8eb2a51759c595a5462b6729ed144f48a18d41 Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 8 Sep 2022 17:11:09 +0200 Subject: [PATCH 08/33] Fix "OTel" case --- ...-elastic-common-schema-in-opentelemetry.md | 36 +++++++++---------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index da965d327..d4b394975 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -1,6 +1,6 @@ # Support Elastic Common Schema in OpenTelemetry -Collaborators: Alolita Sharma (OTEL GC, Apple), Cyrille Le Clerc (Elastic), Daniel Khan (Sentry), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) +Collaborators: Alolita Sharma (OTel GC, Apple), Cyrille Le Clerc (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) ## Introduction @@ -11,35 +11,35 @@ This proposal is to add support for the Elastic Common Schema (ECS) in the OpenT We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions: 1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented by merging ECS fields in OpenTelemetry Semantic Conventions, -2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of Otel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), -3. Actual merge of ECS fields in Otel Semantic Conventions. +2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of OTel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), +3. Actual merge of ECS fields in OTel Semantic Conventions. Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we prefered to first validate the principle and, once consensus is reached, actually define the mapping. ## Motivation -Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTEL) is a great way to accelerate the integration of vendor-created logging and OTEL component logs (ie OTEL Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor-created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTEL correlation to these new signals. +Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTel) is a great way to accelerate the integration of vendor-created logging and OTel component logs (ie OTel Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor-created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTel correlation to these new signals. -Adding the coverage of ECS to OTEL would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTEL Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition. +Adding the coverage of ECS to OTel would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTel Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition. In addition to the use case of structured logs, the maturity of ECS for SIEM (Security Information and Event Management) is a great opportunity for OpenTelemetry to expand its scope to the security use cases. Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type' to identify event types. -We'd like to see different categories of structured logs being well-supported in the [OTEL Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). +We'd like to see different categories of structured logs being well-supported in the [OTel Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). ## Customer Motivation -Adoption of OTEL logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTEL Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated along with open source logs. +Adoption of OTel logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTel Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated along with open source logs. -Customers will benefit from turnkey logs integrations that will be fully recognized by OTEL-compatible observability products and services. +Customers will benefit from turnkey logs integrations that will be fully recognized by OTel-compatible observability products and services. OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)). By expanding the OTel semantic conventions with further namespaces already defined in ECS, a broader coverage of such mappings from different sources can be defined and implemented in the OTel collector. This, for example, includes logs from network appliances (mapping to the `network` and `interface` namespaces in ECS). -The semantic conventions of a log are a challenge. What is a specific component defined in a log and how does it relate to other logs which have the same semantic component defined differently. ECS has already done some heavy-lifting on defining a unified set of semantic conventions which can be adopted in OTEL. +The semantic conventions of a log are a challenge. What is a specific component defined in a log and how does it relate to other logs which have the same semantic component defined differently. ECS has already done some heavy-lifting on defining a unified set of semantic conventions which can be adopted in OTel. OpenTelemetry has the potential to grow exponentially if the data from these other services can be correlated with instrumented code and components. In order to do this, industry stakeholders should leverage a common and standard logging data model which allows for the mapping of these different data types. The OpenTelemetry data protocol can provide this interoperable open standard. This unlocks countless use cases, and ensures that OpenTelemetry can work with other technologies which are not OpenTelemetry compliant. @@ -56,7 +56,7 @@ ECS has the following guiding principles: * ECS favors human readability in order to enable broader adoption as many fields can be understood without having to read up their meaning in the reference, * ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...), - * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by all the events of the source in the way OTEL does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, + * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by all the events of the source in the way OTel does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, * ECS groups fields in namespaces in order to: * Offer consistency and readability, * Enable reusability of namespaces in different contexts, @@ -153,7 +153,7 @@ Example of a Nginx Access Log entry structured with ECS | Dates | uint64 nanoseconds since Unix epoch | date, date_nanos | | Numbers | number | long, double, scaled_float, boolean… | | Objects | uint32, uint64… | object (JSON object), flattened (An entire JSON object as a single field value) | -| Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in Otel).
Note that Otel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | +| Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in OTel).
Note that OTel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | | Binary data | Byte sequence | binary | ## Known Differences @@ -222,13 +222,13 @@ As the markdown code of the tables is hard to read and maintain with very long l - process.cpu.load (not specified but collected by Otel Collector) + process.cpu.load (not specified but collected by OTel Collector)

process.cpu.time (async counter)

-system.cpu.utilization +system.cpu.utilization - host.cpu.usage (scaled_float) with a slightly different measurement than what otel metrics measure + host.cpu.usage (scaled_float) with a slightly different measurement than what OTel metrics measure Note that most metrics have slightly different names and semantics between ECS and OpenTelemetry @@ -239,7 +239,7 @@ As the markdown code of the tables is hard to read and maintain with very long l ### Example an OpenTelemetry Collector Receiver to collect the access logs of a web server -The author of the "Otel Collector Access logs file receiver for web server XXX" would find in the Otel Semantic Convention specifications all the guidance to map the fields of the web server logs, not only the attributes that the Otel Semantic Conventions has specified today for [HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). +The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all the guidance to map the fields of the web server logs, not only the attributes that the OTel Semantic Conventions has specified today for [HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions and enabling richer correlations. ## Alternatives / Discussion @@ -249,13 +249,13 @@ This completeness of the mapping will help the author of the integration to prod Prometheus is a de facto standard for observability metrics and OpenTelemetry already provides full interoperability with the Prometheus ecosystem. -It would be useful to get interoperability between metrics collected by [official Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and their equivalent OpenTelemetry Collector receivers (e.g. Otel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver)). +It would be useful to get interoperability between metrics collected by [official Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/) (e.g. the [Node/system metrics exporter](https://github.com/prometheus/node_exporter) or the [MySQL server exporter](https://github.com/prometheus/mysqld_exporter)) and their equivalent OpenTelemetry Collector receivers (e.g. OTel Collector [Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) or [MySQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver)). Note that one of the challenges with Prometheus metrics naming conventions is that these are implicit conventions defined by each integration author which doesn't enable correlation due to the lack of consistency across integrations. For example, this inconsistency increases the complexity that an end-user has to deal with when configuring and monitoring alerts. Prometheus' conventions are restricted to the style of the name of the metrics (see [Prometheus Metric and label naming](https://prometheus.io/docs/practices/naming/)) but don't specify unified metric names. -## Other areas that need to be addressed by OTEL (the project) +## Other areas that need to be addressed by OTel (the project) -Some areas that need to be addressed in the long run as ECS is integrated into OTEL include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTEL specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). +Some areas that need to be addressed in the long run as ECS is integrated into OTel include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTel specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). From 8632dca2b99760497740c702c5556bea526abcaa Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 8 Sep 2022 17:12:06 +0200 Subject: [PATCH 09/33] Apply suggestions from code review Co-authored-by: Reiley Yang --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index d4b394975..3a441920f 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -14,7 +14,7 @@ We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions: 2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of OTel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), 3. Actual merge of ECS fields in OTel Semantic Conventions. -Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we prefered to first validate the principle and, once consensus is reached, actually define the mapping. +Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we preferred to first validate the principle and, once consensus is reached, actually define the mapping. ## Motivation @@ -24,7 +24,7 @@ Adding the coverage of ECS to OTel would provide guidance to authors of OpenTele In addition to the use case of structured logs, the maturity of ECS for SIEM (Security Information and Event Management) is a great opportunity for OpenTelemetry to expand its scope to the security use cases. -Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using ‘content-type' to identify event types. +Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using 'content-type' to identify event types. We'd like to see different categories of structured logs being well-supported in the [OTel Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). From 2edb6091068036c932c218affca7c6aa439015c9 Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 8 Sep 2022 17:19:49 +0200 Subject: [PATCH 10/33] integrate suggestions --- ...199-support-elastic-common-schema-in-opentelemetry.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 3a441920f..de2b7b111 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -26,7 +26,7 @@ In addition to the use case of structured logs, the maturity of ECS for SIEM (Se Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using 'content-type' to identify event types. -We'd like to see different categories of structured logs being well-supported in the [OTel Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP Access log dashboard benefiting Apache HTTPD, Nginx, and HAProxy). +We'd like to see different categories of structured logs being well-supported in the [OTel Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP access log dashboard benefiting Apache httpd, Nginx, and HAProxy). ## Customer Motivation @@ -126,7 +126,6 @@ Example of a Nginx Access Log entry structured with ECS "id": "04A12D9F-C409-5352-B238-99EA58CAC285", "architecture": "x86_64" } - } ``` @@ -200,7 +199,7 @@ As the markdown code of the tables is hard to read and maintain with very long l SeverityText (string) - log.syslog.severity.name (keyword), log.level (keyword) + log.syslog.severity.name (keyword), log.level (keyword) @@ -223,9 +222,9 @@ As the markdown code of the tables is hard to read and maintain with very long l process.cpu.load (not specified but collected by OTel Collector) -

+
process.cpu.time (async counter) -

+
system.cpu.utilization host.cpu.usage (scaled_float) with a slightly different measurement than what OTel metrics measure From 0d0211af7475646e144003c165c755ef00d69b9b Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Thu, 8 Sep 2022 17:34:46 +0200 Subject: [PATCH 11/33] Rewrap markdown to ease commenting --- ...-elastic-common-schema-in-opentelemetry.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index de2b7b111..46c79590c 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -238,8 +238,15 @@ As the markdown code of the tables is hard to read and maintain with very long l ### Example an OpenTelemetry Collector Receiver to collect the access logs of a web server -The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all the guidance to map the fields of the web server logs, not only the attributes that the OTel Semantic Conventions has specified today for [HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). -This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions and enabling richer correlations. +The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all +the guidance to map the fields of the web server logs, not only the attributes that the OTel Semantic Conventions has specified today for +[HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), +but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) +or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). + +This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs +of other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions +and enabling richer correlations. ## Alternatives / Discussion @@ -257,4 +264,10 @@ Prometheus' conventions are restricted to the style of the name of the metrics ( ## Other areas that need to be addressed by OTel (the project) -Some areas that need to be addressed in the long run as ECS is integrated into OTel include defining the innovation process, providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTel specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). +Some areas that need to be addressed in the long run as ECS is integrated into OTel include defining the innovation process, +providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTel specification incorporates the changes to +accommodate ECS, and a process for handling breaking changes if any (the proposal +[Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) +should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see +[Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , +[Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). From ea7ccc85c19dd09327473c2d6ff44cac5fc3ac07 Mon Sep 17 00:00:00 2001 From: Cyrille Le Clerc Date: Fri, 23 Sep 2022 17:05:21 +0200 Subject: [PATCH 12/33] Fix markdownlint issues --- ...-elastic-common-schema-in-opentelemetry.md | 70 ++++++++----------- 1 file changed, 31 insertions(+), 39 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 46c79590c..caf31aecf 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -2,19 +2,19 @@ Collaborators: Alolita Sharma (OTel GC, Apple), Cyrille Le Clerc (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) - ## Introduction -This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into OTel Semantic Conventions. +This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into OTel Semantic Conventions. ## Proposed process to merge ECS in OpenTelemetry Semantic Conventions We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions: + 1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented by merging ECS fields in OpenTelemetry Semantic Conventions, 2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of OTel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), 3. Actual merge of ECS fields in OTel Semantic Conventions. -Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we preferred to first validate the principle and, once consensus is reached, actually define the mapping. +Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we preferred to first validate the principle and, once consensus is reached, actually define the mapping. ## Motivation @@ -28,14 +28,13 @@ Another significant use case is providing first-class support for Kubernetes app We'd like to see different categories of structured logs being well-supported in the [OTel Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP access log dashboard benefiting Apache httpd, Nginx, and HAProxy). - ## Customer Motivation Adoption of OTel logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTel Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated along with open source logs. Customers will benefit from turnkey logs integrations that will be fully recognized by OTel-compatible observability products and services. -OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)). +OpenTelemetry logging is today mostly structured when instrumentation libraries are used. However, most of the logs which exist today are generated by software, hardware, and cloud services which the user cannot control. OpenTelemetry provides a limited set of "reference integrations" to structure logs: primarily the [OpenTelemetry Collector Kubernetes Events Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver) and an example of a regexp based parsing of Tomcat access logs with OpenTelemetry Collector File Receiver ([here](https://github.com/open-telemetry/opentelemetry-log-collection/blob/30807b96b2f0771e7d11452ebf98fe5e211ed6d7/examples/tomcat/config.yaml#L20)). By expanding the OTel semantic conventions with further namespaces already defined in ECS, a broader coverage of such mappings from different sources can be defined and implemented in the OTel collector. This, for example, includes logs from network appliances (mapping to the `network` and `interface` namespaces in ECS). @@ -43,12 +42,11 @@ The semantic conventions of a log are a challenge. What is a specific component OpenTelemetry has the potential to grow exponentially if the data from these other services can be correlated with instrumented code and components. In order to do this, industry stakeholders should leverage a common and standard logging data model which allows for the mapping of these different data types. The OpenTelemetry data protocol can provide this interoperable open standard. This unlocks countless use cases, and ensures that OpenTelemetry can work with other technologies which are not OpenTelemetry compliant. - ## Background ### What is ECS? -The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic's user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) +The [Elastic Common Schema (ECS)](https://github.com/elastic/ecs) is an open source specification, developed with support from Elastic's user community. ECS defines a common set of fields to be used when storing data in Elasticsearch, such as logs, metrics, and security and audit events. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. Learn more at: [https://www.elastic.co/guide/en/ecs/current/ecs-reference.html](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) The coverage of ECS is very broad including in depth support for logs, security, and network events such as "[logs.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-log.html)" , "[geo.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html)", "[tls.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html)", "[dns.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-dns.html)", or "[vulnerability.* fields](https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html)". @@ -56,21 +54,19 @@ ECS has the following guiding principles: * ECS favors human readability in order to enable broader adoption as many fields can be understood without having to read up their meaning in the reference, * ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...), - * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by all the events of the source in the way OTel does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, + * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by a¬ll the events of the source in the way OTel does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, * ECS groups fields in namespaces in order to: - * Offer consistency and readability, - * Enable reusability of namespaces in different contexts, - * For example, the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces - * Enable extensibility by adding fields to namespaces and adding new namespaces, - * Prevent field name conflicts + * Offer consistency and readability, + * Enable reusability of namespaces in different contexts, + * For example, the "geo" namespace is nested in the "client.geo", "destination.geo", "host.geo" or "threat.indicator.geo" namespaces + * Enable extensibility by adding fields to namespaces and adding new namespaces, + * Prevent field name conflicts * ECS covers a broad spectrum of events with 40+ namespaces including detailed coverage of security and network events. It's much broader than simple logging use cases. - ### Example of a log message structured with ECS: NGINX access logs Example of a Nginx Access Log entry structured with ECS - ```json { "@timestamp":"2020-03-25T09:51:23.000Z", @@ -129,8 +125,6 @@ Example of a Nginx Access Log entry structured with ECS } ``` - - ## Comparison between OpenTelemetry Semantic Conventions for logs and ECS ## Principles @@ -148,12 +142,12 @@ Example of a Nginx Access Log entry structured with ECS | Category | OTel Logs and Event Record (all or a subset of GRPC data types) | ECS Data Types | |---|---|---| -| Text | string | text, match_only_text, keyword constant_keyword, wildcard | -| Dates | uint64 nanoseconds since Unix epoch | date, date_nanos | -| Numbers | number | long, double, scaled_float, boolean… | -| Objects | uint32, uint64… | object (JSON object), flattened (An entire JSON object as a single field value) | -| Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in OTel).
Note that OTel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | -| Binary data | Byte sequence | binary | +| Text | string | text, match_only_text, keyword constant_keyword, wildcard | +| Dates | uint64 nanoseconds since Unix epoch | date, date_nanos | +| Numbers | number | long, double, scaled_float, boolean… | +| Objects | uint32, uint64… | object (JSON object), flattened (An entire JSON object as a single field value) | +| Structured Objects | No complex semantic data type specified for the moment (e.g. string is being used for ip addresses rather than having an "ip" data structure in OTel).
Note that OTel supports arrays and nested objects. | ip, geo_point, geo_shape, version, long_range, date_range, ip_range | +| Binary data | Byte sequence | binary | ## Known Differences @@ -225,7 +219,7 @@ As the markdown code of the tables is hard to read and maintain with very long l
process.cpu.time (async counter)
-system.cpu.utilization +system.cpu.utilization host.cpu.usage (scaled_float) with a slightly different measurement than what OTel metrics measure @@ -234,23 +228,22 @@ As the markdown code of the tables is hard to read and maintain with very long l -## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS +## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS ### Example an OpenTelemetry Collector Receiver to collect the access logs of a web server -The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all -the guidance to map the fields of the web server logs, not only the attributes that the OTel Semantic Conventions has specified today for -[HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), -but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) +The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all +the guidance to map the fields of the web server logs, not only the attributes that the OTel Semantic Conventions has specified today for +[HTTP calls](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.9.0/specification/trace/semantic_conventions/http.md), +but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). -This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs -of other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions +This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs +of other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions and enabling richer correlations. ## Alternatives / Discussion - ### Prometheus Naming Conventions Prometheus is a de facto standard for observability metrics and OpenTelemetry already provides full interoperability with the Prometheus ecosystem. @@ -261,13 +254,12 @@ Note that one of the challenges with Prometheus metrics naming conventions is th Prometheus' conventions are restricted to the style of the name of the metrics (see [Prometheus Metric and label naming](https://prometheus.io/docs/practices/naming/)) but don't specify unified metric names. - ## Other areas that need to be addressed by OTel (the project) -Some areas that need to be addressed in the long run as ECS is integrated into OTel include defining the innovation process, -providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTel specification incorporates the changes to -accommodate ECS, and a process for handling breaking changes if any (the proposal -[Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) -should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see -[Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , +Some areas that need to be addressed in the long run as ECS is integrated into OTel include defining the innovation process, +providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTel specification incorporates the changes to +accommodate ECS, and a process for handling breaking changes if any (the proposal +[Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) +should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see +[Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , [Semantic Conventions for OS Process Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/process-metrics.md)). From c635a85b19259757f62f4aa68f23b8e3a39bb148 Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 18 Oct 2022 11:51:59 +0200 Subject: [PATCH 13/33] Updates the OTEP to reflect ECS as contribution not a merger --- ...-elastic-common-schema-in-opentelemetry.md | 34 ++++++++++++++----- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index caf31aecf..bc1f511a8 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -1,20 +1,34 @@ # Support Elastic Common Schema in OpenTelemetry -Collaborators: Alolita Sharma (OTel GC, Apple), Cyrille Le Clerc (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk) +Collaborators: Alolita Sharma (OTel GC, Apple), Cyrille Le Clerc (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk), Jamie Hynds (Elastic), Martijn Laarman (Elastic), Alexander Wert (Elastic). ## Introduction -This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields. The goal is to merge ECS into OTel Semantic Conventions. +This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields through the donation of complete [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets). The goal is to contribute ECS into OTel Semantic Conventions. -## Proposed process to merge ECS in OpenTelemetry Semantic Conventions +## Proposed process to contribute ECS to OpenTelemetry Semantic Conventions -We propose 3 steps to add support for ECS in OpenTelemetry Semantic Conventions: +This constitutes a contribution where both ECS and Opentelemetry will continue to maintain their own specification and governance bodies. This will minimize breaking changes in both ecosystems. Although existing separately, this OTEP establishes both specifications as extremely closely related specifications and would open up efforts to establish bidirectional synchronization processes. Concretely this would mean: +* An open invitation to OpenTelemetry to participate and review ECS RFC’s +* This could be facilitated through automated github review requests. +* ECS’s RFC process would explicitly call contribution back to OpenTelemetry out as a necessary step. +* Exploring open source community generated conversion tooling as needed. -1. Validation of the principle of adding support for ECS in OpenTelemetry and validation that this support would be implemented by merging ECS fields in OpenTelemetry Semantic Conventions, -2. Validation of the methodology to merge these ECS fields. As there are 40+ ECS namespaces and as there will be few overlaps and maybe needs to evolve some ECS field names to match the vocabulary and conventions of OTel, we have in mind an iterative process tackling namespaces one after the other. We are also interested in clarifying how downstream schemas could be created; We have for example identified the value of having downstream schemas to specify persistence characteristics (see ECS string persistence types match_only_text, keyword constant_keyword, wildcard), -3. Actual merge of ECS fields in OTel Semantic Conventions. +We propose the following steps to add initial support for ECS in OpenTelemetry Semantic Conventions: -Note that we didn't propose in this OTEP the mapping of all ECS fields because this is a substantial effort, we preferred to first validate the principle and, once consensus is reached, actually define the mapping. +1. Open issue to discuss prioritization of [ECS fieldsets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) contribution +2. Open issue for each individual ECS fieldsets for discussion + * See this [Geo fieldset example issue](https://github.com/open-telemetry/opentelemetry-specification/issues/2834) +3. Open draft PR’s to drive implementation (see [Geo example](https://github.com/open-telemetry/opentelemetry-specification/pull/2835)) + * Identifying overlaps/breaking changes. + * Map ECS data types to OpenTelemetry data types. + * Identify high cardinality fields. + * Conflicts would be resolved by: + - Giving preference to existing OpenTelemetry Fields + - Unless a case can be made to update the Experimental OpenTelemetry fields. + Include 2-3 use cases of how these fieldsets are used today as part of the documentation. + +Acceptance of this OTEP registers the intent to kick off this process. ## Motivation @@ -230,6 +244,10 @@ As the markdown code of the tables is hard to read and maintain with very long l ## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS +In most cases they would transparently upgrade to ECS. The main goal of this work is for producers of OpenTelemetry signals (collectors/exporters) to create rich uniform signals. + +The uniformity allows for easier correlation between signals originating from different producers. The richness ensures more options for Root Cause Analysis and reporting. + ### Example an OpenTelemetry Collector Receiver to collect the access logs of a web server The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all From 1debcee7c6d7bd2720f7eb1dc0b6346eea74a391 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Wed, 7 Dec 2022 09:59:24 +0100 Subject: [PATCH 14/33] added clarification on evolution process and dealing with conflicts --- ...-elastic-common-schema-in-opentelemetry.md | 48 ++++++++++++------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index bc1f511a8..89a7010ed 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -1,6 +1,6 @@ # Support Elastic Common Schema in OpenTelemetry -Collaborators: Alolita Sharma (OTel GC, Apple), Cyrille Le Clerc (Elastic), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk), Jamie Hynds (Elastic), Martijn Laarman (Elastic), Alexander Wert (Elastic). +Collaborators: Alexander Wert (Elastic), Jamie Hynds (Elastic), Alolita Sharma (OTel GC, Apple), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk), Jamie Hynds (Elastic) . ## Introduction @@ -8,14 +8,9 @@ This proposal is to add support for the Elastic Common Schema (ECS) in the OpenT ## Proposed process to contribute ECS to OpenTelemetry Semantic Conventions -This constitutes a contribution where both ECS and Opentelemetry will continue to maintain their own specification and governance bodies. This will minimize breaking changes in both ecosystems. Although existing separately, this OTEP establishes both specifications as extremely closely related specifications and would open up efforts to establish bidirectional synchronization processes. Concretely this would mean: -* An open invitation to OpenTelemetry to participate and review ECS RFC’s -* This could be facilitated through automated github review requests. -* ECS’s RFC process would explicitly call contribution back to OpenTelemetry out as a necessary step. -* Exploring open source community generated conversion tooling as needed. - -We propose the following steps to add initial support for ECS in OpenTelemetry Semantic Conventions: +This constitutes a contribution of ECS fieldsets into OpenTelemetry Semantic Conventions whereby both ECS and Opentelemetry will continue to maintain their own specification and governance bodies. This will minimize breaking changes in both ecosystems. Although existing separately, this OTEP establishes both specifications as extremely closely related specifications. +We propose to integrate the contribution of ECS into OpenTelemetry Semantic Conventions as part of a joint review process of existing Semantic Conventions: 1. Open issue to discuss prioritization of [ECS fieldsets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) contribution 2. Open issue for each individual ECS fieldsets for discussion * See this [Geo fieldset example issue](https://github.com/open-telemetry/opentelemetry-specification/issues/2834) @@ -23,10 +18,23 @@ We propose the following steps to add initial support for ECS in OpenTelemetry S * Identifying overlaps/breaking changes. * Map ECS data types to OpenTelemetry data types. * Identify high cardinality fields. - * Conflicts would be resolved by: - - Giving preference to existing OpenTelemetry Fields - - Unless a case can be made to update the Experimental OpenTelemetry fields. - Include 2-3 use cases of how these fieldsets are used today as part of the documentation. + * Include 2-3 use cases of how these fieldsets are used today as part of the documentation + +### Dealing with conflicts +There are fields or fieldsets that conflict between ECS and OpenTelemetry Semantic Conventions. +In some cases it might be possible and reasonable to introduce breaking changes on ECS or OpenTelemetry Semantic Conventions to resolve the conflict. +However, there will also be fields for which it will not be feasible to introduce breaking-changes, both on the ECS side and on the OpenTelemetry Semantic Conventions side. +For these cases, we propose to introduce OpenTelemetry Collector Processors that would provide automated mapping between ECS-specific fields and OpenTelemetry Semantic Conventions atributes. + +### Schema evolution +As described above, the contribution of ECS into OpenTelemetry Semantic Conventions will result in two highly aligned schemas, yet, both specifications will be separate. +Once the initial contribution of ECS into OpenTelemetry Semantic Conventions is completed, an important goal is to maintain an aligned evolution process for both specifications to avoid drift over time. +Therefore, we propose to establish and collaborate on a bidirectional synchronization processes. Concretely this would mean: +* An open invitation to OpenTelemetry to participate and review ECS RFC’s +* This could be facilitated through automated github review requests. +* ECS’s RFC process would explicitly call contribution back to OpenTelemetry out as a necessary step. +* Vice versa, new contributions to the OpenTelemetry Semantic Conventions will be reviewed and adopted by ECS. +* Exploring open source community generated conversion tooling as needed. Acceptance of this OTEP registers the intent to kick off this process. @@ -243,12 +251,14 @@ As the markdown code of the tables is hard to read and maintain with very long l ## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS +The concrete usage of ECS-enriched OpenTelemetry Semantic Conventions Attributes depends on the use case and the fiedset. +In general, OpenTelemetry users would transparently upgrade to ECS and benefit from the alignment of attributes for new use cases. +The main goal of this work is to enable producers of OpenTelemetry signals (collectors/exporters) to create enriched uniform signals for existing and new use cases. +The uniformity allows for easier correlation between signals originating from different producers. The richness ensures more options for Root Cause Analysis, correlation and reporting. -In most cases they would transparently upgrade to ECS. The main goal of this work is for producers of OpenTelemetry signals (collectors/exporters) to create rich uniform signals. +While ECS covers many different use cases and scenarios, in the following, we outline two examples: -The uniformity allows for easier correlation between signals originating from different producers. The richness ensures more options for Root Cause Analysis and reporting. - -### Example an OpenTelemetry Collector Receiver to collect the access logs of a web server +### Example: OpenTelemetry Collector Receiver to collect the access logs of a web server The author of the "OTel Collector Access logs file receiver for web server XXX" would find in the OTel Semantic Convention specifications all the guidance to map the fields of the web server logs, not only the attributes that the OTel Semantic Conventions has specified today for @@ -256,10 +266,14 @@ the guidance to map the fields of the web server logs, not only the attributes t but also attributes for the [User Agent](https://www.elastic.co/guide/en/ecs/current/ecs-user_agent.html) or the [Geo Data](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html). -This completeness of the mapping will help the author of the integration to produce OTel Log Message that will be compatible with access logs +This completeness of the mapping will help the author of the integration to produce OTel Log messages that will be compatible with access logs of other web components (web servers, load balancers, L7 firewalls...) allowing turnkey integration with observability solutions and enabling richer correlations. +### Other Examples +- [Logs with sessions (VPN Logs, Network Access Sessions, RUM sessions, etc.)](https://github.com/elastic/ecs/blob/main/rfcs/text/0004-session.md#usage) +- [Logs from systems processing files](https://www.elastic.co/guide/en/ecs/current/ecs-file.html) + ## Alternatives / Discussion ### Prometheus Naming Conventions From c58f0aa97a4a2524e2f9732ff43f707d92e59f55 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Tue, 13 Dec 2022 10:21:36 +0100 Subject: [PATCH 15/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Georg Pirklbauer --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 89a7010ed..6980e3bc5 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -76,7 +76,7 @@ ECS has the following guiding principles: * ECS favors human readability in order to enable broader adoption as many fields can be understood without having to read up their meaning in the reference, * ECS events include metadata to enable correlations across any dimension (host, data center, docker image, ip address...), - * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by a¬ll the events of the source in the way OTel does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, + * ECS does not differentiate the metadata fields that are specific to each event of the event source and the metadata that is shared by all the events of the source in the way OTel does, which differentiates between Resource Attributes and Log/Span/Metrics Attributes, * ECS groups fields in namespaces in order to: * Offer consistency and readability, * Enable reusability of namespaces in different contexts, From ae0b8e0c70adacecd9ffb006d1a570158785aa55 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Tue, 13 Dec 2022 10:23:26 +0100 Subject: [PATCH 16/33] Updated collaborators --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 6980e3bc5..8e46e3f05 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -1,6 +1,6 @@ # Support Elastic Common Schema in OpenTelemetry -Collaborators: Alexander Wert (Elastic), Jamie Hynds (Elastic), Alolita Sharma (OTel GC, Apple), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran N. (Splunk), Jamie Hynds (Elastic) . +Collaborators: Alexander Wert (Elastic), Jamie Hynds (Elastic), Alolita Sharma (OTel GC, Apple), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran Najaryan (Splunk). ## Introduction From 8dd22ce6fb7e40c925af29f226dd0a7555ca5d0b Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 6 Mar 2023 12:40:43 +0100 Subject: [PATCH 17/33] Updated description to reflect a full donation / merger of ECS and OTel SemConv --- ...-elastic-common-schema-in-opentelemetry.md | 64 ++++++++----------- 1 file changed, 28 insertions(+), 36 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 8e46e3f05..15c4b21f0 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -1,42 +1,34 @@ -# Support Elastic Common Schema in OpenTelemetry - -Collaborators: Alexander Wert (Elastic), Jamie Hynds (Elastic), Alolita Sharma (OTel GC, Apple), Christian Beedgen (Sumo), Jonah Kowall (Logz.io), Tigran Najaryan (Splunk). +# Merge Elastic Common Schema with OpenTelemetry Semantic Conventions ## Introduction -This proposal is to add support for the Elastic Common Schema (ECS) in the OpenTelemetry specification and provide full interoperability for ECS in OpenTelemetry component implementations. We propose to implement this support by enriching OpenTelemetry Semantic Conventions with ECS fields through the donation of complete [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets). The goal is to contribute ECS into OTel Semantic Conventions. - -## Proposed process to contribute ECS to OpenTelemetry Semantic Conventions - -This constitutes a contribution of ECS fieldsets into OpenTelemetry Semantic Conventions whereby both ECS and Opentelemetry will continue to maintain their own specification and governance bodies. This will minimize breaking changes in both ecosystems. Although existing separately, this OTEP establishes both specifications as extremely closely related specifications. - -We propose to integrate the contribution of ECS into OpenTelemetry Semantic Conventions as part of a joint review process of existing Semantic Conventions: -1. Open issue to discuss prioritization of [ECS fieldsets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) contribution -2. Open issue for each individual ECS fieldsets for discussion - * See this [Geo fieldset example issue](https://github.com/open-telemetry/opentelemetry-specification/issues/2834) -3. Open draft PR’s to drive implementation (see [Geo example](https://github.com/open-telemetry/opentelemetry-specification/pull/2835)) - * Identifying overlaps/breaking changes. - * Map ECS data types to OpenTelemetry data types. - * Identify high cardinality fields. - * Include 2-3 use cases of how these fieldsets are used today as part of the documentation - -### Dealing with conflicts -There are fields or fieldsets that conflict between ECS and OpenTelemetry Semantic Conventions. -In some cases it might be possible and reasonable to introduce breaking changes on ECS or OpenTelemetry Semantic Conventions to resolve the conflict. -However, there will also be fields for which it will not be feasible to introduce breaking-changes, both on the ECS side and on the OpenTelemetry Semantic Conventions side. -For these cases, we propose to introduce OpenTelemetry Collector Processors that would provide automated mapping between ECS-specific fields and OpenTelemetry Semantic Conventions atributes. - -### Schema evolution -As described above, the contribution of ECS into OpenTelemetry Semantic Conventions will result in two highly aligned schemas, yet, both specifications will be separate. -Once the initial contribution of ECS into OpenTelemetry Semantic Conventions is completed, an important goal is to maintain an aligned evolution process for both specifications to avoid drift over time. -Therefore, we propose to establish and collaborate on a bidirectional synchronization processes. Concretely this would mean: -* An open invitation to OpenTelemetry to participate and review ECS RFC’s -* This could be facilitated through automated github review requests. -* ECS’s RFC process would explicitly call contribution back to OpenTelemetry out as a necessary step. -* Vice versa, new contributions to the OpenTelemetry Semantic Conventions will be reviewed and adopted by ECS. -* Exploring open source community generated conversion tooling as needed. - -Acceptance of this OTEP registers the intent to kick off this process. +This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry Semantic Conventions (SemConv) and provide full interoperability in OpenTelemetry component implementations. We propose to implement this by aligning the OpenTelemetry Semantic Conventions with [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) and vice versa where feasible. The long-term goal is to achieve convergence of ECS and OTel Semantic Conventions into a single open schema (e.g. OpenTelemetry Common Schema). + +## The Goal +- Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry (with Elastic contributing, participating and co-maintaining the new standard). +- OTel SemConv will adopt ECS in its full scope (except for individual adjustments in detail where needed), including the logging, observability and security domain fields, to make the new schema a true successor of ECS and OTel SemConv. +- Migrate ECS and OTel SemConv users to the new common schema over time and provide utilities to allow the migration to be as easy as possible. + +## Scope and Overlap of ECS and OTel SemConv +ECS and OTel SemConv have some overlap today, but also significant areas of mutually enriching fields. The following diagram illsutrates the different areas: + +

+ +

+ +1. `A`: ECS comes with a rich set of fields that cover broad logging, observability and security use cases. Many fields are additive to the OTel SemConv and would enrich the OTel SemConv without major conflicts. Examples are [Geo information fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html), [Threat Fields](https://www.elastic.co/guide/en/ecs/current/ecs-threat.html), and many others. +2. `B`: Conversely, there are attributes in the OTel SemConv that do not exist in ECS and would be an enrichment to ECS. Examples are the [Messaging semantic conventions](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/messaging/) or technology-specific conventions, such as the [AWS SDK conventions](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/instrumentation/aws-sdk/). +3. `C`: There is some significant area of overlap between ECS and OTel SemConv. The are `C` represents overlappingfields/attributes that are very similar in ECS and OTel SemConv. The field conflicts in `C` can be resolved through simple field renames and simple transformations. +4. `D`: For some of the fields and attributes there will be conflicts that cannot be resolved through simple renaming or transformation and would require introducing breaking changes on ECS or OTel SemConv side for the purpose of merging the schemas. + +## Proposed process to merge ECS with OTel SemConv +The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic's ECS experts will co-maintain and contribute to the new schema and will require the spec approver role in OpenTelemetry for the new schema. + +With the merger there will be different categories of field conflicts between ECS fields and Otel SemConv attributes (as illustrated in the above figure). We expect the areas `A` and `B` to be less controversial and potentially low-handging fruits for an enriched, new schema. + +For the areas `C` and `D` we propose to resolve conflicts through a close collaboration as part of the stabilization initiative of the OTel SemConv. Where feasible, the goal is to align the OTel SemConv attributes as close as possible with the existing, stable ECS fields. Where alignment is not feasible, the goal is to identify ways to address field conflicts through technical transformations and aliasing, to bridge existing fields and formats into the new schema and vice versa (e.g. through OpenTelemetry Collector Processors). + +While realistically truely breaking changes on ECS and OTel SemConv won't be avoidable as part of the merger, they should be the last resort and need to be discussed on a field-by-field basis. ## Motivation From 6d647abcecbbd9a7cbc3106c7711761825604349 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Tue, 7 Mar 2023 07:35:53 +0100 Subject: [PATCH 18/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Reiley Yang --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 15c4b21f0..53a477b26 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -10,7 +10,7 @@ This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry - Migrate ECS and OTel SemConv users to the new common schema over time and provide utilities to allow the migration to be as easy as possible. ## Scope and Overlap of ECS and OTel SemConv -ECS and OTel SemConv have some overlap today, but also significant areas of mutually enriching fields. The following diagram illsutrates the different areas: +ECS and OTel SemConv have some overlap today, but also significant areas of mutually enriching fields. The following diagram illustrates the different areas:

From 354f3984a3dd1bfe6988407ed78eb7cdb0d330df Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Sun, 12 Mar 2023 08:28:22 +0100 Subject: [PATCH 19/33] Update 0199-support-elastic-common-schema-in-opentelemetry.md --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 53a477b26..8e8bcd636 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -5,8 +5,9 @@ This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry Semantic Conventions (SemConv) and provide full interoperability in OpenTelemetry component implementations. We propose to implement this by aligning the OpenTelemetry Semantic Conventions with [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) and vice versa where feasible. The long-term goal is to achieve convergence of ECS and OTel Semantic Conventions into a single open schema (e.g. OpenTelemetry Common Schema). ## The Goal -- Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry (with Elastic contributing, participating and co-maintaining the new standard). -- OTel SemConv will adopt ECS in its full scope (except for individual adjustments in detail where needed), including the logging, observability and security domain fields, to make the new schema a true successor of ECS and OTel SemConv. +- Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry. To kick off this effort, Elastic will nominate several domain experts to join the OpenTelemetry Semantic Convention Approvers. These approvers will be contributing, participating and co-maintaining the new standard. +- OTel SemConv will adopt ECS in its full scope (except for individual adjustments in detail where inevitable), including the logging, observability and security domain fields, to make the new schema a true successor of ECS and OTel SemConv. +- Elastic and OpenTelemetry will coordinate and officially announce the direction of the merger (e.g. through official websites, blog posts, etc.) - Migrate ECS and OTel SemConv users to the new common schema over time and provide utilities to allow the migration to be as easy as possible. ## Scope and Overlap of ECS and OTel SemConv From 14c905e5b0f65cba1864e813e69351f1ae7044e5 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:24:35 -0400 Subject: [PATCH 20/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Trask Stalnaker --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 8e8bcd636..4a07ac683 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -5,7 +5,7 @@ This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry Semantic Conventions (SemConv) and provide full interoperability in OpenTelemetry component implementations. We propose to implement this by aligning the OpenTelemetry Semantic Conventions with [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) and vice versa where feasible. The long-term goal is to achieve convergence of ECS and OTel Semantic Conventions into a single open schema (e.g. OpenTelemetry Common Schema). ## The Goal -- Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry. To kick off this effort, Elastic will nominate several domain experts to join the OpenTelemetry Semantic Convention Approvers. These approvers will be contributing, participating and co-maintaining the new standard. +- Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry. To kick off this effort, Elastic will nominate several domain experts to join the OpenTelemetry Semantic Convention Approvers to help with maintaining the new standard. - OTel SemConv will adopt ECS in its full scope (except for individual adjustments in detail where inevitable), including the logging, observability and security domain fields, to make the new schema a true successor of ECS and OTel SemConv. - Elastic and OpenTelemetry will coordinate and officially announce the direction of the merger (e.g. through official websites, blog posts, etc.) - Migrate ECS and OTel SemConv users to the new common schema over time and provide utilities to allow the migration to be as easy as possible. From f9f544fa49a2745aab2b52615b7bda9a335cda18 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:25:19 -0400 Subject: [PATCH 21/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Trask Stalnaker --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 4a07ac683..ed3003e5b 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -25,7 +25,7 @@ ECS and OTel SemConv have some overlap today, but also significant areas of mutu ## Proposed process to merge ECS with OTel SemConv The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic's ECS experts will co-maintain and contribute to the new schema and will require the spec approver role in OpenTelemetry for the new schema. -With the merger there will be different categories of field conflicts between ECS fields and Otel SemConv attributes (as illustrated in the above figure). We expect the areas `A` and `B` to be less controversial and potentially low-handging fruits for an enriched, new schema. +With the merger there will be different categories of field conflicts between ECS fields and Otel SemConv attributes (as illustrated in the above figure). We expect the areas `A` and `B` to be less controversial and potentially low-hanging fruits for an enriched, new schema. For the areas `C` and `D` we propose to resolve conflicts through a close collaboration as part of the stabilization initiative of the OTel SemConv. Where feasible, the goal is to align the OTel SemConv attributes as close as possible with the existing, stable ECS fields. Where alignment is not feasible, the goal is to identify ways to address field conflicts through technical transformations and aliasing, to bridge existing fields and formats into the new schema and vice versa (e.g. through OpenTelemetry Collector Processors). From 0e32402abe94db69b5a74099a6e5bdabbba377b5 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:25:34 -0400 Subject: [PATCH 22/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Trask Stalnaker --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index ed3003e5b..f61f058bb 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -29,7 +29,7 @@ With the merger there will be different categories of field conflicts between EC For the areas `C` and `D` we propose to resolve conflicts through a close collaboration as part of the stabilization initiative of the OTel SemConv. Where feasible, the goal is to align the OTel SemConv attributes as close as possible with the existing, stable ECS fields. Where alignment is not feasible, the goal is to identify ways to address field conflicts through technical transformations and aliasing, to bridge existing fields and formats into the new schema and vice versa (e.g. through OpenTelemetry Collector Processors). -While realistically truely breaking changes on ECS and OTel SemConv won't be avoidable as part of the merger, they should be the last resort and need to be discussed on a field-by-field basis. +While realistically truly breaking changes on ECS and OTel SemConv won't be avoidable as part of the merger, they should be the last resort and need to be discussed on a field-by-field basis. ## Motivation From 20ef3297f630ed74d8acc3b0bb49a6c12aa21568 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:26:01 -0400 Subject: [PATCH 23/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Trask Stalnaker --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index f61f058bb..7688d8c99 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -33,7 +33,7 @@ While realistically truly breaking changes on ECS and OTel SemConv won't be avoi ## Motivation -Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTel) is a great way to accelerate the integration of vendor-created logging and OTel component logs (ie OTel Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor-created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTel correlation to these new signals. +Adding the Elastic Common Schema (ECS) to OpenTelemetry (OTel) is a great way to accelerate the integration of vendor-created logging and OTel component logs (i.e. OTel Collector Logs Receivers). The goal is to define vendor neutral semantic conventions for most popular types of systems and support vendor-created or open-source components (for example HTTP access logs, network logs, system access/authentication logs) extending OTel correlation to these new signals. Adding the coverage of ECS to OTel would provide guidance to authors of OpenTelemetry Collector Logs Receivers and help establish the OTel Collector as a de facto standard log collector with a well-defined schema to allow for richer data definition. From adc3c56f211d9dac0ff90e52fc48f1bc1b883c92 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:26:18 -0400 Subject: [PATCH 24/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Trask Stalnaker --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 7688d8c99..53f5259a6 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -39,7 +39,7 @@ Adding the coverage of ECS to OTel would provide guidance to authors of OpenTele In addition to the use case of structured logs, the maturity of ECS for SIEM (Security Information and Event Management) is a great opportunity for OpenTelemetry to expand its scope to the security use cases. -Another significant use case is providing first-class support for Kubernetes application logs, system logs as well as application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using 'content-type' to identify event types. +Another significant use case is providing first-class support for Kubernetes application logs, system logs, and application introspection events. We would also like to see support for structured events (e.g. [k8seventsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8seventsreceiver)) and using 'content-type' to identify event types. We'd like to see different categories of structured logs being well-supported in the [OTel Log Data Model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md), presumably through [semantic conventions for log attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-attributes). For example, NGINX access logs and Apache access logs should be processed the same way as structured logs. This would help in trace and metric correlation with such log data as well as it would help grow the ecosystem of curated UIs provided by observability backends and monitoring dashboards (e.g. one single HTTP access log dashboard benefiting Apache httpd, Nginx, and HAProxy). From d06fbca0f4fc118b1be4416dcc27ffff55488741 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:26:37 -0400 Subject: [PATCH 25/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Trask Stalnaker --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 53f5259a6..5111fc067 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -45,7 +45,7 @@ We'd like to see different categories of structured logs being well-supported in ## Customer Motivation -Adoption of OTel logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTel Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated along with open source logs. +Adoption of OTel logs will accelerate greatly if ECS is leveraged as the common standard, using this basis for normalization. OTel Logs adoption will be accelerated by this support. For example, ECS can provide the unified structured format for handling vendor-generated logs along with open source logs. Customers will benefit from turnkey logs integrations that will be fully recognized by OTel-compatible observability products and services. From 0af061add9da5265d271443aca5522542acaf36e Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Mon, 13 Mar 2023 12:31:08 -0400 Subject: [PATCH 26/33] Update 0199-support-elastic-common-schema-in-opentelemetry.md --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 5111fc067..c5fa6efbb 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -23,7 +23,7 @@ ECS and OTel SemConv have some overlap today, but also significant areas of mutu 4. `D`: For some of the fields and attributes there will be conflicts that cannot be resolved through simple renaming or transformation and would require introducing breaking changes on ECS or OTel SemConv side for the purpose of merging the schemas. ## Proposed process to merge ECS with OTel SemConv -The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic's ECS experts will co-maintain and contribute to the new schema and will require the spec approver role in OpenTelemetry for the new schema. +The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic will nominate ECS experts to help with maintainingthe new schema and will require the approver role for the semantic conventions in OpenTelemetry for the new schema. With the merger there will be different categories of field conflicts between ECS fields and Otel SemConv attributes (as illustrated in the above figure). We expect the areas `A` and `B` to be less controversial and potentially low-hanging fruits for an enriched, new schema. From 1e96158c301635e4f70058fc543f41b0d07f1de4 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Tue, 14 Mar 2023 07:36:44 -0400 Subject: [PATCH 27/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Daniel Dyla --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index c5fa6efbb..3d8b15cba 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -244,7 +244,7 @@ As the markdown code of the tables is hard to read and maintain with very long l ## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS -The concrete usage of ECS-enriched OpenTelemetry Semantic Conventions Attributes depends on the use case and the fiedset. +The concrete usage of ECS-enriched OpenTelemetry Semantic Conventions Attributes depends on the use case and the fieldset. In general, OpenTelemetry users would transparently upgrade to ECS and benefit from the alignment of attributes for new use cases. The main goal of this work is to enable producers of OpenTelemetry signals (collectors/exporters) to create enriched uniform signals for existing and new use cases. The uniformity allows for easier correlation between signals originating from different producers. The richness ensures more options for Root Cause Analysis, correlation and reporting. From 8a254ce279d2afabb40fd46684fe952cb088296d Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Tue, 14 Mar 2023 07:40:11 -0400 Subject: [PATCH 28/33] Update 0199-support-elastic-common-schema-in-opentelemetry.md --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 3d8b15cba..18a9a6113 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -282,8 +282,7 @@ Prometheus' conventions are restricted to the style of the name of the metrics ( ## Other areas that need to be addressed by OTel (the project) Some areas that need to be addressed in the long run as ECS is integrated into OTel include defining the innovation process, -providing maintainer rights to Elastic contributors who maintain ECS today, ensuring the OTel specification incorporates the changes to -accommodate ECS, and a process for handling breaking changes if any (the proposal +ensuring the OTel specification incorporates the changes to accommodate ECS, and a process for handling breaking changes if any (the proposal [Define semantic conventions and instrumentation stability #2180](https://github.com/open-telemetry/opentelemetry-specification/pull/2180) should tackle this point). Also, migration of existing naming (e.g. Prometheus exporter) to standardized convention (see [Semantic Conventions for System Metrics](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/system-metrics.md) , From ba2ecc287e2cb5e05dd713a8028faa9cf7a2663f Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Wed, 15 Mar 2023 07:17:07 -0400 Subject: [PATCH 29/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: Armin Ruech --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 18a9a6113..9748f4646 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -23,7 +23,7 @@ ECS and OTel SemConv have some overlap today, but also significant areas of mutu 4. `D`: For some of the fields and attributes there will be conflicts that cannot be resolved through simple renaming or transformation and would require introducing breaking changes on ECS or OTel SemConv side for the purpose of merging the schemas. ## Proposed process to merge ECS with OTel SemConv -The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic will nominate ECS experts to help with maintainingthe new schema and will require the approver role for the semantic conventions in OpenTelemetry for the new schema. +The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic will nominate ECS experts to help with maintaining the new schema and will require the approver role for the semantic conventions in OpenTelemetry for the new schema. With the merger there will be different categories of field conflicts between ECS fields and Otel SemConv attributes (as illustrated in the above figure). We expect the areas `A` and `B` to be less controversial and potentially low-hanging fruits for an enriched, new schema. From 5174ac2280bbeb3a8f8dcce5ae6be6e45f8c9211 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Fri, 17 Mar 2023 12:33:30 -0400 Subject: [PATCH 30/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Co-authored-by: jack-berg <34418638+jack-berg@users.noreply.github.com> --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 9748f4646..abd247dfa 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -53,7 +53,7 @@ OpenTelemetry logging is today mostly structured when instrumentation libraries By expanding the OTel semantic conventions with further namespaces already defined in ECS, a broader coverage of such mappings from different sources can be defined and implemented in the OTel collector. This, for example, includes logs from network appliances (mapping to the `network` and `interface` namespaces in ECS). -The semantic conventions of a log are a challenge. What is a specific component defined in a log and how does it relate to other logs which have the same semantic component defined differently. ECS has already done some heavy-lifting on defining a unified set of semantic conventions which can be adopted in OTel. +The semantic conventions of a log are a challenge. What is a specific component defined in a log and how does it relate to other logs which have the same semantic component defined differently? ECS has already done some heavy-lifting on defining a unified set of semantic conventions which can be adopted in OTel. OpenTelemetry has the potential to grow exponentially if the data from these other services can be correlated with instrumented code and components. In order to do this, industry stakeholders should leverage a common and standard logging data model which allows for the mapping of these different data types. The OpenTelemetry data protocol can provide this interoperable open standard. This unlocks countless use cases, and ensures that OpenTelemetry can work with other technologies which are not OpenTelemetry compliant. From 250797c7eb80743f1c0f2eaebe911726534d1487 Mon Sep 17 00:00:00 2001 From: Alexander Wert Date: Fri, 17 Mar 2023 12:44:01 -0400 Subject: [PATCH 31/33] Update 0199-support-elastic-common-schema-in-opentelemetry.md --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index abd247dfa..d03fef076 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -2,7 +2,7 @@ ## Introduction -This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry Semantic Conventions (SemConv) and provide full interoperability in OpenTelemetry component implementations. We propose to implement this by aligning the OpenTelemetry Semantic Conventions with [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) and vice versa where feasible. The long-term goal is to achieve convergence of ECS and OTel Semantic Conventions into a single open schema (e.g. OpenTelemetry Common Schema). +This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry Semantic Conventions (SemConv) and provide full interoperability in OpenTelemetry component implementations. We propose to implement this by aligning the OpenTelemetry Semantic Conventions with [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) and vice versa where feasible. The long-term goal is to achieve convergence of ECS and OTel Semantic Conventions into a single open schema so that OpenTelemetry Semantic Conventions truly is a successor of the Elastic Common Schema. ## The Goal - Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry. To kick off this effort, Elastic will nominate several domain experts to join the OpenTelemetry Semantic Convention Approvers to help with maintaining the new standard. From 4ed23b65499836b85a4c6659a7e759fb841ca575 Mon Sep 17 00:00:00 2001 From: Reiley Yang Date: Fri, 17 Mar 2023 19:58:32 -0700 Subject: [PATCH 32/33] Update text/0199-support-elastic-common-schema-in-opentelemetry.md Fix typo --- text/0199-support-elastic-common-schema-in-opentelemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index d03fef076..4315d01d1 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -19,7 +19,7 @@ ECS and OTel SemConv have some overlap today, but also significant areas of mutu 1. `A`: ECS comes with a rich set of fields that cover broad logging, observability and security use cases. Many fields are additive to the OTel SemConv and would enrich the OTel SemConv without major conflicts. Examples are [Geo information fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html), [Threat Fields](https://www.elastic.co/guide/en/ecs/current/ecs-threat.html), and many others. 2. `B`: Conversely, there are attributes in the OTel SemConv that do not exist in ECS and would be an enrichment to ECS. Examples are the [Messaging semantic conventions](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/messaging/) or technology-specific conventions, such as the [AWS SDK conventions](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/instrumentation/aws-sdk/). -3. `C`: There is some significant area of overlap between ECS and OTel SemConv. The are `C` represents overlappingfields/attributes that are very similar in ECS and OTel SemConv. The field conflicts in `C` can be resolved through simple field renames and simple transformations. +3. `C`: There is some significant area of overlap between ECS and OTel SemConv. The are `C` represents overlapping fields/attributes that are very similar in ECS and OTel SemConv. The field conflicts in `C` can be resolved through simple field renames and simple transformations. 4. `D`: For some of the fields and attributes there will be conflicts that cannot be resolved through simple renaming or transformation and would require introducing breaking changes on ECS or OTel SemConv side for the purpose of merging the schemas. ## Proposed process to merge ECS with OTel SemConv From 7b8d48f5687e7e398822c1747f542a77c1c0ca3a Mon Sep 17 00:00:00 2001 From: Reiley Yang Date: Sat, 18 Mar 2023 11:16:25 -0700 Subject: [PATCH 33/33] markdownlint --- ...199-support-elastic-common-schema-in-opentelemetry.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/text/0199-support-elastic-common-schema-in-opentelemetry.md b/text/0199-support-elastic-common-schema-in-opentelemetry.md index 4315d01d1..d9e0fb66e 100644 --- a/text/0199-support-elastic-common-schema-in-opentelemetry.md +++ b/text/0199-support-elastic-common-schema-in-opentelemetry.md @@ -5,12 +5,14 @@ This proposal is to merge the Elastic Common Schema (ECS) with the OpenTelemetry Semantic Conventions (SemConv) and provide full interoperability in OpenTelemetry component implementations. We propose to implement this by aligning the OpenTelemetry Semantic Conventions with [ECS FieldSets](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html#ecs-fieldsets) and vice versa where feasible. The long-term goal is to achieve convergence of ECS and OTel Semantic Conventions into a single open schema so that OpenTelemetry Semantic Conventions truly is a successor of the Elastic Common Schema. ## The Goal + - Long-term, ECS and OTel SemConv will converge into one open standard that is maintained by OpenTelemetry. To kick off this effort, Elastic will nominate several domain experts to join the OpenTelemetry Semantic Convention Approvers to help with maintaining the new standard. - OTel SemConv will adopt ECS in its full scope (except for individual adjustments in detail where inevitable), including the logging, observability and security domain fields, to make the new schema a true successor of ECS and OTel SemConv. - Elastic and OpenTelemetry will coordinate and officially announce the direction of the merger (e.g. through official websites, blog posts, etc.) - Migrate ECS and OTel SemConv users to the new common schema over time and provide utilities to allow the migration to be as easy as possible. ## Scope and Overlap of ECS and OTel SemConv + ECS and OTel SemConv have some overlap today, but also significant areas of mutually enriching fields. The following diagram illustrates the different areas:

@@ -20,14 +22,15 @@ ECS and OTel SemConv have some overlap today, but also significant areas of mutu 1. `A`: ECS comes with a rich set of fields that cover broad logging, observability and security use cases. Many fields are additive to the OTel SemConv and would enrich the OTel SemConv without major conflicts. Examples are [Geo information fields](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html), [Threat Fields](https://www.elastic.co/guide/en/ecs/current/ecs-threat.html), and many others. 2. `B`: Conversely, there are attributes in the OTel SemConv that do not exist in ECS and would be an enrichment to ECS. Examples are the [Messaging semantic conventions](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/messaging/) or technology-specific conventions, such as the [AWS SDK conventions](https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/instrumentation/aws-sdk/). 3. `C`: There is some significant area of overlap between ECS and OTel SemConv. The are `C` represents overlapping fields/attributes that are very similar in ECS and OTel SemConv. The field conflicts in `C` can be resolved through simple field renames and simple transformations. -4. `D`: For some of the fields and attributes there will be conflicts that cannot be resolved through simple renaming or transformation and would require introducing breaking changes on ECS or OTel SemConv side for the purpose of merging the schemas. +4. `D`: For some of the fields and attributes there will be conflicts that cannot be resolved through simple renaming or transformation and would require introducing breaking changes on ECS or OTel SemConv side for the purpose of merging the schemas. ## Proposed process to merge ECS with OTel SemConv + The process of merging ECS with OTel SemConv will take time and we propose to do it as part of the stabilization effort for OTel SemConv. During that period and also for a significant period after the merger (sunset period), Elastic will continue to support ECS as a schema. However, further evolution of ECS will happen on the basis of the new, common schema. Elastic will nominate ECS experts to help with maintaining the new schema and will require the approver role for the semantic conventions in OpenTelemetry for the new schema. With the merger there will be different categories of field conflicts between ECS fields and Otel SemConv attributes (as illustrated in the above figure). We expect the areas `A` and `B` to be less controversial and potentially low-hanging fruits for an enriched, new schema. -For the areas `C` and `D` we propose to resolve conflicts through a close collaboration as part of the stabilization initiative of the OTel SemConv. Where feasible, the goal is to align the OTel SemConv attributes as close as possible with the existing, stable ECS fields. Where alignment is not feasible, the goal is to identify ways to address field conflicts through technical transformations and aliasing, to bridge existing fields and formats into the new schema and vice versa (e.g. through OpenTelemetry Collector Processors). +For the areas `C` and `D` we propose to resolve conflicts through a close collaboration as part of the stabilization initiative of the OTel SemConv. Where feasible, the goal is to align the OTel SemConv attributes as close as possible with the existing, stable ECS fields. Where alignment is not feasible, the goal is to identify ways to address field conflicts through technical transformations and aliasing, to bridge existing fields and formats into the new schema and vice versa (e.g. through OpenTelemetry Collector Processors). While realistically truly breaking changes on ECS and OTel SemConv won't be avoidable as part of the merger, they should be the last resort and need to be discussed on a field-by-field basis. @@ -244,6 +247,7 @@ As the markdown code of the tables is hard to read and maintain with very long l ## How would OpenTelemetry users practically use the new OpenTelemetry Semantic Conventions Attributes brought by ECS + The concrete usage of ECS-enriched OpenTelemetry Semantic Conventions Attributes depends on the use case and the fieldset. In general, OpenTelemetry users would transparently upgrade to ECS and benefit from the alignment of attributes for new use cases. The main goal of this work is to enable producers of OpenTelemetry signals (collectors/exporters) to create enriched uniform signals for existing and new use cases. @@ -264,6 +268,7 @@ of other web components (web servers, load balancers, L7 firewalls...) allowing and enabling richer correlations. ### Other Examples + - [Logs with sessions (VPN Logs, Network Access Sessions, RUM sessions, etc.)](https://github.com/elastic/ecs/blob/main/rfcs/text/0004-session.md#usage) - [Logs from systems processing files](https://www.elastic.co/guide/en/ecs/current/ecs-file.html)