Change fields.ecs.yml to ECS 1.11.0 #27107

leehinman · 2021-07-28T15:02:57Z

What does this PR do?

fields.ecs.yml to ECS 1.11.0-dev
bump ecs.version in configs for modules that don't require changes.

Why is it important?

keep up to date with ECS changes
allows modules to use new ECS fields

Checklist

My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd filebeat && mage pythonIntegTest
cd x-pack/filebeat && mage pythonIntegTest

Related issues

Relates [ECS] Upgrade modules to ECS 1.11 #26967

elasticmachine · 2021-07-28T15:24:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-16T23:30:00.134+0000
Duration: 233 min 23 sec
Commit: 0df0ce5

Test stats 🧪

Test	Results
Failed	0
Passed	53106
Skipped	5320
Total	58426

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53106
Skipped	5320
Total	58426

elasticmachine · 2021-07-29T03:27:08Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-07-30T11:38:11Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beats_ecs_1.11 upstream/beats_ecs_1.11
git merge upstream/master
git push upstream beats_ecs_1.11

andrewkroh

I'm curious how many new fields does this release add?

(My concern is that we add new fields to every beat with each ECS update (even if they are not utilized). This increases the mappings stored in cluster state and grows the response sizes in Kibana from the fields API.)

mergify · 2021-08-13T08:28:58Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beats_ecs_1.11 upstream/beats_ecs_1.11
git merge upstream/master
git push upstream beats_ecs_1.11

andrewstucki

Some comments/questions.

I'm thinking we should probably add orchestrator data to the processors as well:
- https:/elastic/beats/tree/master/libbeat/processors/add_kubernetes_metadata
- https:/elastic/beats/tree/master/x-pack/libbeat/processors/add_nomad_metadata
For the docker autodiscover/processor do you know if there's a way via Docker's API to detect if you're running in Swarm? If so maybe we should consider adding there as well? If it's too much of a heavy lift maybe we can add that in later as a separate enhancement though.

andrewstucki · 2021-08-13T13:40:10Z

libbeat/autodiscover/providers/kubernetes/orchestrator.go

+
+ nodeMap, err := kubemeta.GetValue(kind)
+ if err == nil {
+ node, _ := nodeMap.(common.MapStr)


Does this need to be cast/nil checked? If you want to do an unconditional cast, I'd remove the _ -- but assuming the above GetValue call returns a nil you probably want to be more explicit about the cast and nil checking here.

Some comments/questions.

* I'm thinking we should probably add orchestrator data to the processors as well: * https:/elastic/beats/tree/master/libbeat/processors/add_kubernetes_metadata * https:/elastic/beats/tree/master/x-pack/libbeat/processors/add_nomad_metadata

Added

* For the docker autodiscover/processor do you know if there's a way via Docker's API to detect if you're running in Swarm? If so maybe we should consider adding there as well? If it's too much of a heavy lift maybe we can add that in later as a separate enhancement though.

It looks like you might be able to "inspect swarm", and know if you are in a swarm or not. But I'm thinking a separate enhancement might be best.

Does this need to be cast/nil checked? If you want to do an unconditional cast, I'd remove the _ -- but assuming the above GetValue call returns a nil you probably want to be more explicit about the cast and nil checking here.

changed this so we don't get intermediate maps, cleaner all around, don't know what I was thinking.

andrewstucki · 2021-08-13T13:43:30Z

libbeat/autodiscover/providers/kubernetes/pod.go

@@ -435,14 +437,16 @@ func (p *pod) podEvent(flag string, pod *kubernetes.Pod, ports common.MapStr, in
 if len(namespaceAnnotations) != 0 {
 kubemeta["namespace_annotations"] = namespaceAnnotations
 }
+ orchestrator := genOrchestratorFields(kubemeta, "pod")


Is there some guidance around the orchestrator.resource.type field? Just making sure it makes sense to put pod here, the examples in the docs all have service and not sure if there was a desire to make the language orchestrator agnostic or not.

Kubernetes autodiscover provider can discover different kinds of resources, depending on the resource setting. When resource: service is used, it will collect events from services, and then I think it is fine to use orchestrator.resource.type: service there, but here it is discovering pods and containers, so I think that it is ok to use pod.

andrewstucki · 2021-08-13T13:44:43Z

x-pack/libbeat/autodiscover/providers/nomad/nomad.go

+ if err != nil {
+ return common.MapStr{}
+ }
+ nomad, _ := nomadMap.(common.MapStr)


Same as previous casting comments on unconditional v. checked casts.

andrewstucki · 2021-08-13T13:44:56Z

x-pack/libbeat/autodiscover/providers/nomad/nomad.go

+
+ taskMap, err := nomad.GetValue("task")
+ if err == nil {
+ task, _ := taskMap.(common.MapStr)


Same as previous casting comments on unconditional v. checked casts.

leehinman · 2021-08-13T15:17:21Z

I'm curious how many new fields does this release add?

(My concern is that we add new fields to every beat with each ECS update (even if they are not utilized). This increases the mappings stored in cluster state and grows the response sizes in Kibana from the fields API.)

I counted 792 fields in ECS 1.10.0 fields.ecs.yml and 1207 in 1.11.0, for a difference of 415 fields added. That was after flattening the field names.

Diff is:
1.10-1.11_diff.txt

mergify · 2021-08-13T16:53:59Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beats_ecs_1.11 upstream/beats_ecs_1.11
git merge upstream/master
git push upstream beats_ecs_1.11

leehinman · 2021-08-13T20:48:45Z

@jsoriano & @kaiyan-sheng could you take a look at the changes to kubernetes & nomad autodiscover and add_metadata processors in this PR. It is a start to adding the orchestrator fields, and I'd really like a sanity check.

jsoriano

Reviewed the nomad and kubernetes parts, added some comments about the types used, and also about possible interactions with the code that adds orchestrator.cluster.name/url fields in some cases.

As there can be more discussion in these parts, I would suggest to move them to a different PR if you want to move forward the upgrade of ECS.

ccing @MichaelKatsoulis for the kubernetes changes too.

jsoriano · 2021-08-16T07:43:32Z

x-pack/libbeat/autodiscover/providers/nomad/nomad.go

+ taskName, err := meta.GetValue("nomad.task.name")
+ if err == nil {
+ orchestrator.Put("resource.name", taskName)
+ orchestrator.Put("resource.type", "task")


Events are generated at the allocation level, that is one level more specific than tasks (job -> task -> allocation), so I wonder if we should use allocations here instead of tasks.

For example, with filebeat, for a job with an nginx task in a task group with multiple instances I have events with:

"nomad": { "allocation": { "id": "afb658cf-5a5f-aeef-8fa7-0755906919d1", "status": "running", "name": "redis.server[0]" }, ... "job": { "name": "redis", "type": "service" }, "task": { "name": "nginx", "service": { ...

And:

"nomad": { "allocation": { "status": "running", "name": "redis.server[1]", "id": "feeddc6b-6f28-0cb6-8351-9dfec5dfcaac" }, "job": { "name": "redis", "type": "service" }, "region": "global", "task": { "name": "nginx", "service": { ...

jsoriano · 2021-08-16T07:49:19Z

x-pack/libbeat/processors/add_nomad_metadata/nomad.go

+ if err == nil {
+ orchestrator.Put("namespace", namespace)
+ }
+


Interestingly add_nomad_metadata doesn't collect task-level metadata, this should be probably improved. But it collects allocation metadata, I think we should use this information if we want to fill type and name (in coherence with my other comment).

An example of the nomad metadata collected by add_nomad_metadata:

"nomad": { "region": "global", "allocation": { "id": "feeddc6b-6f28-0cb6-8351-9dfec5dfcaac", "status": "running", "name": "redis.server[1]" }, "job": { "name": "redis", "type": "service" }, "namespace": "default", "datacenter": [ "dc1" ] },

jsoriano · 2021-08-16T08:01:23Z

libbeat/autodiscover/providers/kubernetes/pod.go

@@ -383,6 +383,7 @@ func (p *pod) containerPodEvents(flag string, pod *kubernetes.Pod, c *containerI
 if len(namespaceAnnotations) != 0 {
 kubemeta["namespace_annotations"] = namespaceAnnotations
 }
+ orchestrator := genOrchestratorFields(kubemeta, "pod")


Events here are emitted at the container level, should we use the container fields instead?

jsoriano · 2021-08-16T08:04:51Z

libbeat/autodiscover/providers/kubernetes/pod.go

+ "meta": meta,
+ "orchestrator": orchestrator,


I think that to actually enrich the events, the new fields should be set in meta. I think that the other fields are used only in autodiscover (for templates, conditions and so on) but don't end up in the final documents. (But not 100% sure, I always get confused by that).

jsoriano · 2021-08-16T08:14:14Z

libbeat/autodiscover/providers/kubernetes/pod.go

@@ -435,14 +437,16 @@ func (p *pod) podEvent(flag string, pod *kubernetes.Pod, ports common.MapStr, in
 if len(namespaceAnnotations) != 0 {
 kubemeta["namespace_annotations"] = namespaceAnnotations
 }
+ orchestrator := genOrchestratorFields(kubemeta, "pod")


Kubernetes autodiscover provider can discover different kinds of resources, depending on the resource setting. When resource: service is used, it will collect events from services, and then I think it is fine to use orchestrator.resource.type: service there, but here it is discovering pods and containers, so I think that it is ok to use pod.

jsoriano · 2021-08-16T08:23:11Z

libbeat/processors/add_kubernetes_metadata/kubernetes.go

 event.Fields.DeepUpdate(kubeMeta)
+ event.Fields.DeepUpdate(orchestrator)


When beats are run in GKE, add_cloud_metadata adds the orchestrator.cluster.name/url fields, but I think that only if the orchestrator field doesn't exist (unless fields are flattened when they reach here). We will have to check how well both things play together.

Same problem may exist with the autodiscover provider.

For context, the name/url fields were added in #26056 and #26438.

@jsoriano Thank you for reviewing. And you've verified my suspicion that I should pull the orchestrator parts out and do that is a separate PR.

mergify · 2021-08-16T22:06:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b beats_ecs_1.11 upstream/beats_ecs_1.11
git merge upstream/master
git push upstream beats_ecs_1.11

- fields.ecs.yml to ECS 1.11.0-dev - bump ecs.version in configs for modules that don't require changes. Relates elastic#26967

- kubernetes - nomad

will do as separate PR

* Change fields.ecs.yml to ECS 1.11.0 - fields.ecs.yml to ECS 1.11.0 - bump ecs.version in configs for modules that don't require changes. Relates #26967 (cherry picked from commit 5869e1f)

* Change fields.ecs.yml to ECS 1.11.0 - fields.ecs.yml to ECS 1.11.0 - bump ecs.version in configs for modules that don't require changes. Relates #26967 (cherry picked from commit 5869e1f) Co-authored-by: Lee Hinman <[email protected]>

simitt · 2021-08-31T15:03:56Z

This PR breaks compatibility with older ES versions (tested with apm-server), which are not supporting flattened type. According to the support matrix, 7.x beats are supposed to support the whole range of ES 7.x.

update:

flattened is supported in ES >= 7.3
the PR also introduces additional constant_keyword supported in ES >= 7.7 (constant_keywords were introduced in 7.14 already update libbeat fields.ecs.yml file and ecsVersion to 1.10.0 #26121).

leehinman · 2021-08-31T19:09:36Z

This PR breaks compatibility with older ES versions (tested with apm-server), which are not supporting flattened type. According to the support matrix, 7.x beats are supposed to support the whole range of ES 7.x.

Thanks for catching this.

leehinman added enhancement Filebeat Filebeat Team:Security-External Integrations backport-v7.15.0 Automated backport with mergify labels Jul 28, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 28, 2021

leehinman force-pushed the beats_ecs_1.11 branch 5 times, most recently from 43f02f5 to 36cc6f9 Compare July 28, 2021 22:22

leehinman marked this pull request as ready for review July 29, 2021 03:27

leehinman requested a review from a team as a code owner July 29, 2021 03:27

leehinman added the libbeat label Jul 29, 2021

andrewkroh approved these changes Aug 3, 2021

View reviewed changes

leehinman force-pushed the beats_ecs_1.11 branch 4 times, most recently from f5d3efa to 2b8a519 Compare August 13, 2021 03:58

leehinman force-pushed the beats_ecs_1.11 branch from 2b8a519 to 784ac35 Compare August 13, 2021 13:49

andrewstucki reviewed Aug 13, 2021

View reviewed changes

leehinman changed the title ~~Change fields.ecs.yml to ECS 1.11.0-dev~~ Change fields.ecs.yml to ECS 1.11.0 Aug 13, 2021

leehinman force-pushed the beats_ecs_1.11 branch from 784ac35 to e951ec8 Compare August 13, 2021 20:42

jsoriano reviewed Aug 16, 2021

View reviewed changes

leehinman force-pushed the beats_ecs_1.11 branch 3 times, most recently from 6c45afc to a811822 Compare August 16, 2021 20:34

andrewkroh approved these changes Aug 16, 2021

View reviewed changes

leehinman added 4 commits August 16, 2021 18:26

Change fields.ecs.yml to ECS 1.11.0

ef83a72

- fields.ecs.yml to ECS 1.11.0-dev - bump ecs.version in configs for modules that don't require changes. Relates elastic#26967

add orchestrator fields to autodiscover providers

c092b8e

- kubernetes - nomad

fields update

86d1973

remove orchestrator changes

0df0ce5

will do as separate PR

leehinman force-pushed the beats_ecs_1.11 branch from a811822 to 0df0ce5 Compare August 16, 2021 23:29

leehinman merged commit 5869e1f into elastic:master Aug 17, 2021

mergify bot mentioned this pull request Aug 17, 2021

[7.x](backport #27107) Change fields.ecs.yml to ECS 1.11.0 #27421

Merged

leehinman deleted the beats_ecs_1.11 branch August 17, 2021 21:15

simitt mentioned this pull request Aug 31, 2021

ingest/pipeline: fix compat with old Elasticsearch elastic/apm-server#5876

Merged

1 task

andrewkroh mentioned this pull request Aug 31, 2021

[Filebeat] Module incompatibility with older ES/Kibana versions #26629

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change fields.ecs.yml to ECS 1.11.0 #27107

Change fields.ecs.yml to ECS 1.11.0 #27107

leehinman commented Jul 28, 2021 •

edited

Loading

elasticmachine commented Jul 28, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Jul 29, 2021

mergify bot commented Jul 30, 2021

andrewkroh left a comment

mergify bot commented Aug 13, 2021

andrewstucki left a comment

andrewstucki Aug 13, 2021

leehinman Aug 13, 2021

leehinman Aug 13, 2021

andrewstucki Aug 13, 2021

jsoriano Aug 16, 2021

andrewstucki Aug 13, 2021

andrewstucki Aug 13, 2021

leehinman commented Aug 13, 2021

mergify bot commented Aug 13, 2021

leehinman commented Aug 13, 2021

jsoriano left a comment

jsoriano Aug 16, 2021

jsoriano Aug 16, 2021 •

edited

Loading

jsoriano Aug 16, 2021

jsoriano Aug 16, 2021

jsoriano Aug 16, 2021

jsoriano Aug 16, 2021

leehinman Aug 16, 2021

mergify bot commented Aug 16, 2021

simitt commented Aug 31, 2021 •

edited

Loading

leehinman commented Aug 31, 2021

		event.Fields.DeepUpdate(kubeMeta)
		event.Fields.DeepUpdate(orchestrator)

Change fields.ecs.yml to ECS 1.11.0 #27107

Change fields.ecs.yml to ECS 1.11.0 #27107

Conversation

leehinman commented Jul 28, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Jul 28, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Jul 29, 2021

mergify bot commented Jul 30, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

mergify bot commented Aug 13, 2021

andrewstucki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leehinman commented Aug 13, 2021

mergify bot commented Aug 13, 2021

leehinman commented Aug 13, 2021

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano Aug 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Aug 16, 2021

simitt commented Aug 31, 2021 • edited Loading

leehinman commented Aug 31, 2021

leehinman commented Jul 28, 2021 •

edited

Loading

elasticmachine commented Jul 28, 2021 •

edited by jenkins-beats-ci bot

Loading

jsoriano Aug 16, 2021 •

edited

Loading

simitt commented Aug 31, 2021 •

edited

Loading