ECS on by default for Logstash 8, again (#13391) (#13416)

* ecs: report pipeline's ECS-compatibility with INFO at startup

Because the pipeline-level setting `pipeline.ecs_compatibility` affects the
default behaviour of nearly every plugin in the pipeline, an INFO-level log
message will provide useful hints, especially to our users who upgrade to
Logstash 8 without first reading the breaking changes docs.

For example, when we have two pipelines `old` and `new` whose `pipeline.ecs_compatibility` is `disabled` and `v8` respectively, we would get the following log messages:

> ~~~
> [2021-11-04T18:43:21,810][INFO ][logstash.javapipeline    ] Pipeline `old` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
> [2021-11-04T18:43:21,817][INFO ][logstash.javapipeline    ] Pipeline `new` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
> ~~~

* ecs: make v8 the default for 8.0

* ecs: `pipeline.ecs_compatibility` defaults to `v8`

Related: #11623

* doc: temporarily remove deep link from breaking changes doc to fix build

(cherry picked from commit c3e498a)

config/logstash.yml

@@ -70,17 +70,15 @@
 # available to plugins that implement an ECS Compatibility mode for use with
 # the Elastic Common Schema.
 # Possible values are:
-# - disabled (default)
+# - disabled
 # - v1
-# - v8
-# CAVEAT: use of this configuration for anything other than `disabled`
-# is considered BETA until the General Availability of
-# Logstash 8.0.0. As we continue to release updated plugins with ECS-Compatibility
-# modes, opting into them at a pipeline or process level will cause you to
-# automatically consume breaking changes with each upgrade, which may change the
-# shape of data your pipeline produces.
-#
-# pipeline.ecs_compatibility: disabled
+# - v8 (default)
+# Pipelines defined before Logstash 8 operated without ECS in mind. To ensure a
+# migrated pipeline continues to operate as it did before your upgrade, opt-OUT
+# of ECS for the individual pipeline in its `pipelines.yml` definition. Setting
+# it here will set the default for _all_ pipelines, including new ones.
+#
+# pipeline.ecs_compatibility: v8
 #
 # ------------ Pipeline Configuration Settings --------------
 #

docs/static/breaking-changes.asciidoc

@@ -46,6 +46,15 @@ Users who need to use a version other than the bundled JDK should set the value
 of `LS_JAVA_HOME` to the path of their preferred JDK. 
 The value of `JAVA_HOME` will be ignored.
 
+[discrete]
+[[bc-ecs-compatibility]]
+===== ECS compatibility is now on by default
+Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema.
+This mode is controlled individually with each plugin's `ecs_compatibility` option, which defaults to the value of the Logstash `pipeline.ecs_compatibility` setting.
+In Logstash 8, this compatibility mode will be on-by-default for all pipelines. https:/elastic/logstash/issues/11623[#11623]
+
+If you wish to _lock in_ a pipeline's behaviour from Logstash 7.x before upgrading to Logstash 8, you can set `pipeline.ecs_compatibility: disabled` to its definition in `pipelines.yml` (or globally in `logstash.yml`).
+
 [discrete]
 [[bc-ruby-engine]]
 ===== Ruby Execution Engine removed

docs/static/ecs-compatibility.asciidoc

@@ -2,7 +2,7 @@
 === ECS in Logstash
 
 // LS8 will ship with ECS v8, but until ECS v8 is ready we rely on ECS v1 as an approximation.
-:ls8-ecs-major-version: v1
+:ls8-ecs-major-version: v8
 
 The {ecs-ref}/index.html[Elastic Common Schema (ECS)] is an open source specification, developed with support from the Elastic user community.
 ECS defines a common set of fields to be used for storing event data, such as logs and metrics, in {es}.
@@ -15,25 +15,16 @@ Many plugins implement an ECS-compatibility mode, which causes them to produce a
 
 Any plugin that supports this mode will also have an `ecs_compatibility` option, which allows you to configure which mode the individual plugin instance should operate in.
 If left unspecified for an individual plugin, the pipeline's `pipeline.ecs_compatibility` setting will be observed.
-This allows you to configure plugins to use ECS -- or to lock in your existing non-ECS behavior.
+This allows you to configure plugins to use a specific version of ECS or to use their legacy non-ECS behavior.
 
 ECS compatibility modes do not prevent you from explicitly configuring a plugin in a manner that conflicts with ECS.
 Instead, they ensure that _implicit_ configuration avoids conflicts.
 
-NOTE: Until {ls} 8.0 and the final 7.x are released, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported.
- As we continue to release plugins with ECS compatibility modes, having this flag set will cause even patch-level upgrades to _automatically_ consume breaking changes in the upgraded plugins, changing the shape of data the plugin produces.
-
-ifeval::["{ls8-ecs-major-version}"!="v8"]
-NOTE: ECS `v8` will be available alongside the GA release of {ls} 8.0.0, and will be available at or before the final minor release of {ls} 7.
- We expect the scope of breaking changes in ECS 8 to be limited.
- We are https:/elastic/ecs/issues/839[tracking progress toward ECS v8] in a GitHub issue.
-endif::[]
-
 [[ecs-configuration]]
 ===== Configuring ECS
 
-ECS will be on by default in a future release of {ls}, but you can begin using it now by configuring individual plugins with `ecs_compatibility`.
-You can also "lock in" the existing non-ECS behavior for an entire pipeline to ensure its behavior doesn't change when you perform future upgrades.
+In {ls} 8, all plugins are run in ECS compatibility {ls8-ecs-major-version} mode by default, but you can opt out at the plugin, pipeline, or system level to maintain legacy behavior.
+This can be helpful if you have very complex pipelines that were defined pre-ECS, to allow you to either upgrade them or to avoid doing so independently of your {ls} 8.x upgrade.
 
 ====== Specific plugin instance
 
@@ -73,6 +64,8 @@ If you wish to provide a specific default value for `ecs_compatibility` to _all_
 This setting will be used unless overridden by a specific plugin instance.
 If unspecified for an individual pipeline, the global value will be used.
 
+For example, setting `pipeline.ecs_compatibility: disabled` for a pipeline _locks in_ that pipeline's pre-{ls} 8 behavior.
+
 [source,yaml,subs="attributes"]
 -----
 - pipeline.id: my-legacy-pipeline
@@ -83,9 +76,6 @@ If unspecified for an individual pipeline, the global value will be used.
  pipeline.ecs_compatibility: {ls8-ecs-major-version}
 -----
 
-NOTE: Until the General Availability of {ls} 8.0.0 that coincides with the final minor release of {ls} 7, any value for `pipeline.ecs_compatibility` other than `disabled` is considered BETA and unsupported because it will produce undesirable consequences when performing upgrades.
- As we continue to release updated plugins with ECS-Compatibility modes, opting into them at a pipeline or process level will cause the affected plugins to silently and automatically consume breaking changes with each upgrade, which may change the shape of data your pipeline produces.
-
 [[ecs-configuration-all]]
 ====== All plugins in all pipelines
 

logstash-core/lib/logstash/agent.rb

@@ -65,14 +65,6 @@ def initialize(settings = LogStash::SETTINGS, source_loader = nil)
  # Generate / load the persistent uuid
  id
 
- if @settings.set?('pipeline.ecs_compatibility')
- ecs_compatibility_value = settings.get('pipeline.ecs_compatibility')
- if ecs_compatibility_value != 'disabled'
- logger.warn("Setting `pipeline.ecs_compatibility` given as `#{ecs_compatibility_value}`; " +
- "values other than `disabled` are currently considered BETA and may have unintended consequences when upgrading minor versions of Logstash.")
- end
- end
-
  # Initialize, but do not start the webserver.
  @webserver = LogStash::WebServer.from_settings(@logger, self, settings)
 

logstash-core/lib/logstash/environment.rb

@@ -60,7 +60,7 @@ module Environment
  Setting::Boolean.new("pipeline.plugin_classloaders", false),
  Setting::Boolean.new("pipeline.separate_logs", false),
  Setting::CoercibleString.new("pipeline.ordered", "auto", true, ["auto", "true", "false"]),
- Setting::CoercibleString.new("pipeline.ecs_compatibility", "disabled", true, %w(disabled v1 v8)),
+ Setting::CoercibleString.new("pipeline.ecs_compatibility", "v8", true, %w(disabled v1 v8)),
  Setting.new("path.plugins", Array, []),
  Setting::NullableString.new("interactive", nil, false),
  Setting::Boolean.new("config.debug", false),

logstash-core/lib/logstash/java_pipeline.rb

@@ -72,6 +72,10 @@ def initialize(pipeline_config, namespaced_metric = nil, agent = nil)
  # is by design and necessary for the wait_until_started semantic
  @finished_run = Concurrent::AtomicBoolean.new(false)
 
+ @logger.info(I18n.t('logstash.pipeline.effective_ecs_compatibility',
+ :pipeline_id => pipeline_id,
+ :ecs_compatibility => settings.get('pipeline.ecs_compatibility')))
+
  @thread = nil
  end # def initialize
 

logstash-core/lib/logstash/plugins/ecs_compatibility_support.rb

@@ -17,11 +17,6 @@ def ecs_compatibility
  pipeline_settings = pipeline && pipeline.settings
  pipeline_settings ||= LogStash::SETTINGS
 
- if !pipeline_settings.set?('pipeline.ecs_compatibility')
- deprecation_logger.deprecated("Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. " +
- "To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.")
- end
-
  pipeline_settings.get_value('pipeline.ecs_compatibility').to_sym
  end
  end

logstash-core/locales/en.yml

@@ -45,6 +45,9 @@ en:
  %{plugin} output plugin: setting 'workers => %{worker_count}' is not
  supported by this plugin. I will continue working as if you had not set
  this setting. Reason: %{message}
+ effective_ecs_compatibility: >-
+ Pipeline `%{pipeline_id}` is configured with `pipeline.ecs_compatibility: %{ecs_compatibility}` setting.
+ All plugins in this pipeline will default to `ecs_compatibility => %{ecs_compatibility}` unless explicitly configured otherwise.
  plugin:
  deprecated_milestone: >-
  %{plugin} plugin is using the 'milestone' method to declare the version

logstash-core/spec/logstash/plugin_spec.rb

@@ -461,16 +461,9 @@ def register; end
  end
 
  context 'and pipeline-level setting is not specified' do
- it 'emits a deprecation warning about using the default which may change' do
- instance.ecs_compatibility
-
- expect(deprecation_logger_stub).to have_received(:deprecated) do |message|
- expect(message).to include("Relying on default value of `pipeline.ecs_compatibility`")
- end
- end
- it 'returns `disabled`' do
+ it 'returns `v8`' do
  # Default value of `pipeline.ecs_compatibility`
- expect(instance.ecs_compatibility).to eq(:disabled)
+ expect(instance.ecs_compatibility).to eq(:v8)
  end
  end
  end