ECS on by default for Logstash 8, again #13391

yaauie · 2021-11-04T19:48:16Z

Release notes

[breaking] The pipeline.ecs_compatibility setting now defaults to v8, causing all plugins that implement an ECS compatibility mode to default to operating with ecs_compatibility => v8.

What does this PR do?

Effectively reverts #13080, making the default ECS compatibility mode v8 for Logstash 8, and adds INFO-level logging for each pipeline to indicate in which mode its plugins will be run.

Why is it important/What is the impact to the user?

This the final step of a large and ongoing effort to provide better support for the Elastic Common Schema, detailed in #11623. In #13080 we had back-tracked making the breaking change out of an abundance of caution, but have since decided to move forward with it.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files (and/or docker env variables)
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

There are a number of plugins that ship with Logstash and have ECS modes, that do NOT yet have modes for v8. Because the scope of changes for v1->v8 is so small, I think that all of these can provide a v8 implementation that simply aliases the existing v1 implementation.

geoip filter: ecs: add v8 preview with v1 implementation logstash-plugins/logstash-filter-geoip#193
grok filter: ecs: add v8 preview using v1 implementation logstash-plugins/logstash-filter-grok#175 (alias, but log warning; need to validate against v1->v8 scope changes in finer detail)
clone filter: ECS: add v8 support as alias for v1 logstash-plugins/logstash-filter-clone#27
syslog_pri filter: ecs: add v8 preview with v1 implementation logstash-plugins/logstash-filter-syslog_pri#10
useragent filter: ecs: add v8 preview with v1 implementation logstash-plugins/logstash-filter-useragent#76
syslog input: ecs: add v8 alias to v1 implementation logstash-plugins/logstash-input-syslog#68

How to test this PR locally

bin/logstash -e ''

Observe an info-level log indicating that the pipeline is being run in ECS-compatibility mode.
Observe no deprecation log from the stdin input indicating that the default ecs_compatibility may change
Process one event, observe that the structure is ECS-compliant ([host][hostname])

Related issues

Resolves #11623
Reverts #13080

Because the pipeline-level setting `pipeline.ecs_compatibility` affects the default behaviour of nearly every plugin in the pipeline, an INFO-level log message will provide useful hints, especially to our users who upgrade to Logstash 8 without first reading the breaking changes docs. For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages: > ~~~ > [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise. > [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. > ~~~

Related: elastic#11623

yaauie · 2021-11-04T20:22:09Z

docs/static/breaking-changes.asciidoc

+[discrete]
+[[bc-ecs-compatibility]]
+===== ECS compatibility is now on by default
+Many plugins can now be run in a mode that avoids implicit conflict with the <<ecs-ls,Elastic Common Schema>>.


@karenzone this breaks the docs build, but I don't expect it to because we define ecs-ls here: https:/yaauie/logstash/blame/ecs-on-by-default-again/docs/static/ecs-compatibility.asciidoc#L1 -- Am I missing something?

Looks reasonable and builds fine locally. Re-running docs-ci while I investigate further.

@elasticmachine run elasticsearch-ci/docs

Hmmmm... Docs-ci is wonky and not running on command. Definitely some weirdness going on. Still investigating.

@elasticmachine run elasticsearch-ci/docs

We've both re-triggered docs-CI several times, and it continues to fail. When I build locally it works fine, and therefore I can't recreate the issue to troubleshoot it. I agree with your proposal to remove the x-ref for now. We'll be doing more work in this file before release, and will have ample opportunity to add it back.

yaauie · 2021-11-05T19:25:15Z

jenkins test this again please

kares

👍 so far so good

tested locally with stdout as well as ES output and all seems to work (including data-streams) 🎉

one minor thing is that we have the "type" => "stdin" set from the stdin input, not sure whether it got missed or stayed on purpose (in ecs_compatibility mode) in the plugin, type has no conflicts with ECS schema atm.

CI 🔴 failures are mostly due pending v8 plugin releases.

kares · 2021-11-08T07:05:14Z

logstash-core/lib/logstash/environment.rb

@@ -60,7 +60,7 @@ module Environment
 Setting::Boolean.new("pipeline.plugin_classloaders", false),
 Setting::Boolean.new("pipeline.separate_logs", false),
 Setting::CoercibleString.new("pipeline.ordered", "auto", true, ["auto", "true", "false"]),
- Setting::CoercibleString.new("pipeline.ecs_compatibility", "disabled", true, %w(disabled v1 v8)),
+ Setting::CoercibleString.new("pipeline.ecs_compatibility", "v8", true, %w(disabled v1 v8)),


yaauie · 2021-11-11T20:39:25Z

jenkins test this again please

yaauie · 2021-11-15T13:01:15Z

Jenkins test this again, please

…gain

karenzone

Left some comments inline for your consideration.

karenzone · 2021-11-17T00:14:58Z

docs/static/breaking-changes.asciidoc

+[discrete]
+[[bc-ecs-compatibility]]
+===== ECS compatibility is now on by default
+Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema.


Suggested change

Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema.

Many plugins can now be run in a mode that avoids implicit conflict with the {ecs-ref}[Elastic Common Schema (ECS)].

karenzone · 2021-11-17T00:16:02Z

docs/static/breaking-changes.asciidoc

+This mode is controlled individually with each plugin's `ecs_compatibility` option, which defaults to the value of the Logstash `pipeline.ecs_compatibility` setting.
+In Logstash 8, this compatibility mode will be on-by-default for all pipelines. https:/elastic/logstash/issues/11623[#11623]
+
+If you wish to _lock in_ a pipeline's behaviour from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`).


Suggested change

If you wish to _lock in_ a pipeline's behaviour from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`).

If you wish to _lock in_ a pipeline's behavior from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`).

The Elastic standard is US English spellings.

karenzone · 2021-11-17T00:20:03Z

docs/static/ecs-compatibility.asciidoc

-ECS will be on by default in a future release of {ls}, but you can begin using it now by configuring individual plugins with `ecs_compatibility`.
-You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn't change when you perform future upgrades.
+In {ls} 8, all plugins are run in ECS compatibility {ls8-ecs-major-version} mode by default, but you can opt out at the plugin, pipeline, or system level to maintain legacy behavior.
+This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.


Suggested change

This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.

These options can be helpful if you have very complex pipelines that were defined pre-ECS.

You can upgrade them or avoid doing so independently of your {ls} 8.x upgrade.

kares

LGTM, tested again locally with the generator input - using ECS by default ✌️

Not propagating ecs_compatibility setting from input -> codec might be decremental to the user xp but is expected (users should opt-out on the whole pipeline instead of individual plugins) and out of scope, maybe something to consider as an improvement later.

* ecs: report pipeline's ECS-compatibility with INFO at startup Because the pipeline-level setting `pipeline.ecs_compatibility` affects the default behaviour of nearly every plugin in the pipeline, an INFO-level log message will provide useful hints, especially to our users who upgrade to Logstash 8 without first reading the breaking changes docs. For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages: > ~~~ > [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise. > [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. > ~~~ * ecs: make v8 the default for 8.0 * ecs: `pipeline.ecs_compatibility` defaults to `v8` Related: elastic#11623 * doc: temporarily remove deep link from breaking changes doc to fix build (cherry picked from commit c3e498a)

* ecs: report pipeline's ECS-compatibility with INFO at startup Because the pipeline-level setting `pipeline.ecs_compatibility` affects the default behaviour of nearly every plugin in the pipeline, an INFO-level log message will provide useful hints, especially to our users who upgrade to Logstash 8 without first reading the breaking changes docs. For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages: > ~~~ > [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise. > [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. > ~~~ * ecs: make v8 the default for 8.0 * ecs: `pipeline.ecs_compatibility` defaults to `v8` Related: #11623 * doc: temporarily remove deep link from breaking changes doc to fix build (cherry picked from commit c3e498a)

yaauie added 3 commits November 4, 2021 18:53

ecs: make v8 the default for 8.0

25a019f

ecs: pipeline.ecs_compatibility defaults to v8

24743fd

Related: elastic#11623

yaauie added enhancement breaking change labels Nov 4, 2021

yaauie requested review from kares and karenzone November 4, 2021 19:48

roaksoax added the status:work-in-progress label Nov 4, 2021

yaauie commented Nov 4, 2021

View reviewed changes

kares reviewed Nov 8, 2021

View reviewed changes

yaauie added 2 commits November 16, 2021 15:29

doc: temporarily remove deep link from breaking changes doc to fix build

12c4680

Merge remote-tracking branch 'upstream/main' into ecs-on-by-default-a…

ec17079

…gain

yaauie marked this pull request as ready for review November 16, 2021 16:04

roaksoax added status:needs-review and removed status:work-in-progress labels Nov 16, 2021

karenzone reviewed Nov 17, 2021

View reviewed changes

kares approved these changes Nov 17, 2021

View reviewed changes

roaksoax added status:approved and removed status:needs-review labels Nov 17, 2021

yaauie merged commit c3e498a into elastic:main Nov 17, 2021

yaauie deleted the ecs-on-by-default-again branch November 17, 2021 21:17

yaauie mentioned this pull request Nov 17, 2021

Backport PR #13391 to 8.0: ECS on by default for Logstash 8, again #13416

Merged

11 tasks

yaauie added the v8.0.0 label Nov 17, 2021

roaksoax removed the status:approved label Nov 17, 2021

kares added the ecs label Nov 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECS on by default for Logstash 8, again #13391

ECS on by default for Logstash 8, again #13391

yaauie commented Nov 4, 2021 •

edited

Loading

yaauie Nov 4, 2021

karenzone Nov 4, 2021 •

edited

Loading

karenzone Nov 4, 2021

karenzone Nov 4, 2021

karenzone Nov 4, 2021

karenzone Nov 16, 2021

yaauie commented Nov 5, 2021

kares left a comment

kares Nov 8, 2021

yaauie commented Nov 11, 2021

yaauie commented Nov 15, 2021

karenzone left a comment

karenzone Nov 17, 2021

karenzone Nov 17, 2021

karenzone Nov 17, 2021

karenzone Nov 17, 2021

kares left a comment

	Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema.
	Many plugins can now be run in a mode that avoids implicit conflict with the {ecs-ref}[Elastic Common Schema (ECS)].

	If you wish to _lock in_ a pipeline's behaviour from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`).
	If you wish to _lock in_ a pipeline's behavior from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`).

	This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.
	These options can be helpful if you have very complex pipelines that were defined pre-ECS.
	You can upgrade them or avoid doing so independently of your {ls} 8.x upgrade.

ECS on by default for Logstash 8, again #13391

ECS on by default for Logstash 8, again #13391

Conversation

yaauie commented Nov 4, 2021 • edited Loading

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Choose a reason for hiding this comment

karenzone Nov 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaauie commented Nov 5, 2021

kares left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaauie commented Nov 11, 2021

yaauie commented Nov 15, 2021

karenzone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kares left a comment

Choose a reason for hiding this comment

yaauie commented Nov 4, 2021 •

edited

Loading

karenzone Nov 4, 2021 •

edited

Loading