Reduce default template logging at startup

[jakelandis]

When Elasticsearch starts up it is kinda noisy w.r.t. the index templates added by default :

[2021-12-06T17:12:09,577][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.ml-state] for index patterns [.ml-state*]
[2021-12-06T17:12:09,647][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.ml-anomalies-] for index patterns [.ml-anomalies-*]
[2021-12-06T17:12:09,685][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.ml-notifications-000002] for index patterns [.ml-notifications-000002]
[2021-12-06T17:12:09,722][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.ml-stats] for index patterns [.ml-stats-*]
[2021-12-06T17:12:09,749][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [synthetics-settings]
[2021-12-06T17:12:09,783][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [synthetics-mappings]
[2021-12-06T17:12:09,818][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [data-streams-mappings]
[2021-12-06T17:12:09,847][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [metrics-mappings]
[2021-12-06T17:12:09,878][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [logs-mappings]
[2021-12-06T17:12:09,902][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [logs-settings]
[2021-12-06T17:12:09,927][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [metrics-settings]
[2021-12-06T17:12:09,968][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [ilm-history] for index patterns [ilm-history-5*]
[2021-12-06T17:12:10,002][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.watch-history-13] for index patterns [.watcher-history-13*]
[2021-12-06T17:12:10,032][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.slm-history] for index patterns [.slm-history-5*]
[2021-12-06T17:12:10,059][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [.deprecation-indexing-settings]
[2021-12-06T17:12:10,088][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding component template [.deprecation-indexing-mappings]
[2021-12-06T17:12:10,114][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2021-12-06T17:12:10,151][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2021-12-06T17:12:10,180][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2021-12-06T17:12:10,213][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]
[2021-12-06T17:12:10,254][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2021-12-06T17:12:10,286][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [synthetics] for index patterns [synthetics-*-*]
[2021-12-06T17:12:10,317][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [logs] for index patterns [logs-*-*]
[2021-12-06T17:12:10,345][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [metrics] for index patterns [metrics-*-*]
[2021-12-06T17:12:10,378][INFO ][o.e.c.m.MetadataIndexTemplateService] [link] adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.*]

Most of this logging should be turned into a debug logging and ideally only print out 1 or 2 INFO messages that encompass the high level info.

related: #81356 and #81397

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

Perhaps tangentially related to #77505

[martijnvg]

The tricky bit is that the put component and index template APIs produce this info log.
I suspect when these APIs are invoked normally we would like an INFO log?
But when invoked from our IndexTemplateRegistry infrastructure we don't want an INFO log.

Maybe we can add a field to PutComponentTemplateAction.Request and
PutComposableIndexTemplateAction.Request classes (not accessible via rest) that controls whether
the logging occurs (or sets the logging to DEBUG level)? Then in IndexTemplateRegistry
we can emit a single info log per implementation (ilm, stack, ml etc.)?

[DaveCTurner]

I do agree that these lines at startup are excessively noisy but I would rather we didn't drop the info about changes to ILM policies from the INFO level logs. They are sometimes invaluable in tracking what happened in support cases.

Could we batch these changes into a single update which results in a single log line? Could we enhance the startup lifecycle so that the stuff that users need to see is delayed until these things are complete?

[DaveCTurner]

(Kinda relates #77505 too)

[martijnvg]

All implementations of IndexTemplateRegistry (except FleetTemplateRegistry (i guess that can be changed)) only install templates / ilm policies when running on the elected master. Perhaps IndexTemplateRegistry can just directly interact with the node services (e.g. MetadataIndexTemplateService) that install these resources. We can then change these services to do batching without changing the corresponding APIs. (this could be the ground work that a bulk like api is built upon).