-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ECS on by default for Logstash 8, again #13391
Changes from 3 commits
07e950e
25a019f
24743fd
12c4680
ec17079
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -34,6 +34,15 @@ Here are the breaking changes for 8.0. | |||||
[[bc-core]] | ||||||
==== Changes in Logstash Core | ||||||
|
||||||
[discrete] | ||||||
[[bc-ecs-compatibility]] | ||||||
===== ECS compatibility is now on by default | ||||||
Many plugins can now be run in a mode that avoids implicit conflict with the <<ecs-ls,Elastic Common Schema>>. | ||||||
This mode is controlled individually with each plugin's `ecs_compatibility` option, which defaults to the value of the Logstash `pipeline.ecs_compatibility` setting. | ||||||
In Logstash 8, this compatibility mode will be on-by-default for all pipelines. https:/elastic/logstash/issues/11623[#11623] | ||||||
|
||||||
If you wish to _lock in_ a pipeline's behaviour from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`). | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The Elastic standard is US English spellings. |
||||||
|
||||||
[discrete] | ||||||
[[bc-ruby-engine]] | ||||||
===== Ruby Execution Engine removed | ||||||
|
Original file line number | Diff line number | Diff line change | ||||||
---|---|---|---|---|---|---|---|---|
|
@@ -2,7 +2,7 @@ | |||||||
=== ECS in Logstash | ||||||||
|
||||||||
// LS8 will ship with ECS v8, but until ECS v8 is ready we rely on ECS v1 as an approximation. | ||||||||
:ls8-ecs-major-version: v1 | ||||||||
:ls8-ecs-major-version: v8 | ||||||||
|
||||||||
The {ecs-ref}/index.html[Elastic Common Schema (ECS)] is an open source specification, developed with support from the Elastic user community. | ||||||||
ECS defines a common set of fields to be used for storing event data, such as logs and metrics, in {es}. | ||||||||
|
@@ -15,25 +15,16 @@ Many plugins implement an ECS-compatibility mode, which causes them to produce a | |||||||
|
||||||||
Any plugin that supports this mode will also have an `ecs_compatibility` option, which allows you to configure which mode the individual plugin instance should operate in. | ||||||||
If left unspecified for an individual plugin, the pipeline's `pipeline.ecs_compatibility` setting will be observed. | ||||||||
This allows you to configure plugins to use ECS -- or to lock in your existing non-ECS behavior. | ||||||||
This allows you to configure plugins to use a specific version of ECS or to use their legacy non-ECS behavior. | ||||||||
|
||||||||
ECS compatibility modes do not prevent you from explicitly configuring a plugin in a manner that conflicts with ECS. | ||||||||
Instead, they ensure that _implicit_ configuration avoids conflicts. | ||||||||
|
||||||||
NOTE: Until {ls} 8.0 and the final 7.x are released, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported. | ||||||||
As we continue to release plugins with ECS compatibility modes, having this flag set will cause even patch-level upgrades to _automatically_ consume breaking changes in the upgraded plugins, changing the shape of data the plugin produces. | ||||||||
|
||||||||
ifeval::["{ls8-ecs-major-version}"!="v8"] | ||||||||
NOTE: ECS `v8` will be available alongside the GA release of {ls} 8.0.0, and will be available at or before the final minor release of {ls} 7. | ||||||||
We expect the scope of breaking changes in ECS 8 to be limited. | ||||||||
We are https:/elastic/ecs/issues/839[tracking progress toward ECS v8] in a GitHub issue. | ||||||||
endif::[] | ||||||||
|
||||||||
[[ecs-configuration]] | ||||||||
===== Configuring ECS | ||||||||
|
||||||||
ECS will be on by default in a future release of {ls}, but you can begin using it now by configuring individual plugins with `ecs_compatibility`. | ||||||||
You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn't change when you perform future upgrades. | ||||||||
In {ls} 8, all plugins are run in ECS compatibility {ls8-ecs-major-version} mode by default, but you can opt out at the plugin, pipeline, or system level to maintain legacy behavior. | ||||||||
This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade. | ||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||
|
||||||||
====== Specific plugin instance | ||||||||
|
||||||||
|
@@ -73,6 +64,8 @@ If you wish to provide a specific default value for `ecs_compatibility` to _all_ | |||||||
This setting will be used unless overridden by a specific plugin instance. | ||||||||
If unspecified for an individual pipeline, the global value will be used. | ||||||||
|
||||||||
For example, setting `pipeline.ecs_compatibility: disabled` for a pipeline _locks in_ that pipeline's pre-{ls} 8 behavior. | ||||||||
|
||||||||
[source,yaml,subs="attributes"] | ||||||||
----- | ||||||||
- pipeline.id: my-legacy-pipeline | ||||||||
|
@@ -83,9 +76,6 @@ If unspecified for an individual pipeline, the global value will be used. | |||||||
pipeline.ecs_compatibility: {ls8-ecs-major-version} | ||||||||
----- | ||||||||
|
||||||||
NOTE: Until the General Availability of {ls} 8.0.0 that coincides with the final minor release of {ls} 7, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported because it will produce undesirable consequences when performing upgrades. | ||||||||
As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause the affected plugins to silently and automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces. | ||||||||
|
||||||||
[[ecs-configuration-all]] | ||||||||
====== All plugins in all pipelines | ||||||||
|
||||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -60,7 +60,7 @@ module Environment | |
Setting::Boolean.new("pipeline.plugin_classloaders", false), | ||
Setting::Boolean.new("pipeline.separate_logs", false), | ||
Setting::CoercibleString.new("pipeline.ordered", "auto", true, ["auto", "true", "false"]), | ||
Setting::CoercibleString.new("pipeline.ecs_compatibility", "disabled", true, %w(disabled v1 v8)), | ||
Setting::CoercibleString.new("pipeline.ecs_compatibility", "v8", true, %w(disabled v1 v8)), | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ☯️ |
||
Setting.new("path.plugins", Array, []), | ||
Setting::NullableString.new("interactive", nil, false), | ||
Setting::Boolean.new("config.debug", false), | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@karenzone this breaks the docs build, but I don't expect it to because we define
ecs-ls
here: https:/yaauie/logstash/blame/ecs-on-by-default-again/docs/static/ecs-compatibility.asciidoc#L1 -- Am I missing something?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks reasonable and builds fine locally. Re-running docs-ci while I investigate further.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@elasticmachine run elasticsearch-ci/docs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmmm... Docs-ci is wonky and not running on command. Definitely some weirdness going on. Still investigating.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@elasticmachine run elasticsearch-ci/docs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've both re-triggered docs-CI several times, and it continues to fail. When I build locally it works fine, and therefore I can't recreate the issue to troubleshoot it. I agree with your proposal to remove the x-ref for now. We'll be doing more work in this file before release, and will have ample opportunity to add it back.