Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add system network fields #5436

Merged
merged 10 commits into from
Jun 16, 2021
Merged

Add system network fields #5436

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b58556b
add system network fields
jalvz Jun 10, 2021
010925b
code review
jalvz Jun 14, 2021
01c456a
Update model/system_test.go
jalvz Jun 14, 2021
c2add24
Update processor/otel/metadata.go
jalvz Jun 14, 2021
f9433e7
liiiiint
jalvz Jun 15, 2021
d69e467
add changelog
jalvz Jun 15, 2021
d36f30f
Merge branch 'master' into mobile-fields
jalvz Jun 15, 2021
6145741
Merge branch 'master' into mobile-fields
jalvz Jun 15, 2021
d0d5b06
Merge branch 'master' into mobile-fields
jalvz Jun 16, 2021
8a24fa5
Merge branch 'master' into mobile-fields
jalvz Jun 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions apmpackage/apm/data_stream/app_metrics/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,26 @@
type: long
description: Current data collection period for this event in milliseconds.
unit: ms
- name: network.carrier.icc
type: keyword
description: |
ISO country code, eg. US
- name: network.carrier.mcc
type: keyword
description: |
Mobile country code
- name: network.carrier.mnc
type: keyword
description: |
Mobile network code
- name: network.carrier.name
type: keyword
description: |
Carrier name, eg. Vodafone, T-Mobile, etc.
- name: network.connection_type
type: keyword
description: |
Cellular network technology, eg. 4G
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not certain about the naming. ECS has a definition for network and one for host with some host.network fields. None of them contain what we need. This PR introduces network.carrier.* fields, that are not (yet) in ECS, but also not aligned with the otel proposal net.host.carrier.*.

@jalvz did you have a conversation with the ECS folks where they would see these fields fit in? I am a bit worried to just remove the host part, that is in the otel proposal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already a top level host field, I didn't think it would make much sense to duplicate the field inside network to refer to the same host?
network is already defined as top level field in ECS (https://www.elastic.co/guide/en/ecs/current/ecs-network.html),

I sort of followed #491, btw.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ECS says (about the network fieldset):

The network is defined as the communication path over which a host or network event happens.

The network.* fields should be populated with details about the network activity associated with an event.

Seems to me it makes sense to go under network

Copy link
Contributor Author

@jalvz jalvz Jun 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So let me ask:

  • host and network.host would refer to the same host, or different ones?
  • If different: which is what? If same: why duplicate a field we already have?
  • What exactly says in the ECS network definition that dictates there should be a host field inside network? (sorry I can't see that in the quote)
  • If we go with network.host, why don't we apply the same reasoning elsewhere? Eg., a process normally runs on a host, yet there is no process.host; rather 2 top fields (conceptually related).
  • If we go with network.host, how do you justify that in the context of other fields? We could have eg. network.host.connection_type="wifi" and network.name="Guest wifi" - why connection_type and name would not be at the same level?
  • What does it mean, conceptually, network.host.carrier?: a host has a carrier? a carrier runs on a host?...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some offline discussion, network.carrier.* is where the new fields fit best related to ECS, and net.host.carrier.* was the prefered nesting in the open-telemetry proposal; so 👍 on the changes.

- name: observer.listening
type: keyword
description: |
Expand Down
20 changes: 20 additions & 0 deletions apmpackage/apm/data_stream/error_logs/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,26 @@
type: keyword
description: |
Kubernetes Pod UID
- name: network.carrier.icc
type: keyword
description: |
ISO country code, eg. US
- name: network.carrier.mcc
type: keyword
description: |
Mobile country code
- name: network.carrier.mnc
type: keyword
description: |
Mobile network code
- name: network.carrier.name
type: keyword
description: |
Carrier name, eg. Vodafone, T-Mobile, etc.
- name: network.connection_type
type: keyword
description: |
Cellular network technology, eg. 4G
- name: observer.listening
type: keyword
description: |
Expand Down
20 changes: 20 additions & 0 deletions apmpackage/apm/data_stream/internal_metrics/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,26 @@
type: long
description: Current data collection period for this event in milliseconds.
unit: ms
- name: network.carrier.icc
type: keyword
description: |
ISO country code, eg. US
- name: network.carrier.mcc
type: keyword
description: |
Mobile country code
- name: network.carrier.mnc
type: keyword
description: |
Mobile network code
- name: network.carrier.name
type: keyword
description: |
Carrier name, eg. Vodafone, T-Mobile, etc.
- name: network.connection_type
type: keyword
description: |
Cellular network technology, eg. 4G
- name: observer.listening
type: keyword
description: |
Expand Down
20 changes: 20 additions & 0 deletions apmpackage/apm/data_stream/profile_metrics/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,26 @@
type: keyword
description: |
Kubernetes Pod UID
- name: network.carrier.icc
type: keyword
description: |
ISO country code, eg. US
- name: network.carrier.mcc
type: keyword
description: |
Mobile country code
- name: network.carrier.mnc
type: keyword
description: |
Mobile network code
- name: network.carrier.name
type: keyword
description: |
Carrier name, eg. Vodafone, T-Mobile, etc.
- name: network.connection_type
type: keyword
description: |
Cellular network technology, eg. 4G
- name: observer.listening
type: keyword
description: |
Expand Down
20 changes: 20 additions & 0 deletions apmpackage/apm/data_stream/traces/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,26 @@
type: keyword
description: |
Kubernetes Pod UID
- name: network.carrier.icc
type: keyword
description: |
ISO country code, eg. US
- name: network.carrier.mcc
type: keyword
description: |
Mobile country code
- name: network.carrier.mnc
type: keyword
description: |
Mobile network code
- name: network.carrier.name
type: keyword
description: |
Carrier name, eg. Vodafone, T-Mobile, etc.
- name: network.connection_type
type: keyword
description: |
Cellular network technology, eg. 4G
- name: observer.listening
type: keyword
description: |
Expand Down
15 changes: 15 additions & 0 deletions apmpackage/apm/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ Traces are written to `traces-apm.*` indices.
|kubernetes.pod.name|Kubernetes pod name|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|kubernetes.pod.uid|Kubernetes Pod UID|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|labels|A flat mapping of user-defined labels with string, boolean or number values.|object| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
|network.carrier.icc|ISO country code, eg. US|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.mcc|Mobile country code|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.mnc|Mobile network code|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.name|Carrier name, eg. Vodafone, T-Mobile, etc.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.connection\_type|Cellular network technology, eg. 4G|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|observer.hostname|Hostname of the APM Server.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
|observer.listening|Address the server is listening on.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|observer.type|The type will be set to \`apm-server\`.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
Expand Down Expand Up @@ -446,6 +451,11 @@ Metrics are written to `metrics-apm.app.*`, `metrics-apm.internal.*`, and `metri
|labels|A flat mapping of user-defined labels with string, boolean or number values.|object| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
|metricset.name|Name of the set of metrics.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|metricset.period|Current data collection period for this event in milliseconds.|long| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.icc|ISO country code, eg. US|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.mcc|Mobile country code|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.mnc|Mobile network code|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.name|Carrier name, eg. Vodafone, T-Mobile, etc.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.connection\_type|Cellular network technology, eg. 4G|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|observer.hostname|Hostname of the APM Server.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
|observer.listening|Address the server is listening on.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|observer.type|The type will be set to \`apm-server\`.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
Expand Down Expand Up @@ -631,6 +641,11 @@ Logs are written to `logs-apm.error.*` indices.
|kubernetes.pod.name|Kubernetes pod name|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|kubernetes.pod.uid|Kubernetes Pod UID|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|labels|A flat mapping of user-defined labels with string, boolean or number values.|object| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
|network.carrier.icc|ISO country code, eg. US|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.mcc|Mobile country code|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.mnc|Mobile network code|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.carrier.name|Carrier name, eg. Vodafone, T-Mobile, etc.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|network.connection\_type|Cellular network technology, eg. 4G|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|observer.hostname|Hostname of the APM Server.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
|observer.listening|Address the server is listening on.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-no.png) |
|observer.type|The type will be set to \`apm-server\`.|keyword| ![](https://doc-icons.s3.us-east-2.amazonaws.com/icon-yes.png) |
Expand Down
1 change: 1 addition & 0 deletions changelogs/head.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ https:/elastic/apm-server/compare/7.13\...master[View commits]
* Upgrade Go to 1.16.4 {pull}5381[5381]
* Add units to metric fields {pull}5395[5395]
* Add support for adjusting OTel event timestamps using `telemetry.sdk.elastic_export_timestamp` {pull}5433[5433]
* Add support for OpenTelemetry labels describing mobile connectivity {pull}5436[5436]

[float]
==== Deprecated
Loading