[Telemetry] [schema] Define schema for the data that is reported by the telemetry plugin

[afharo]

We currently have a way to define the schema that is returned by the Usage Collectors.

They usually end up under stack_stats.kibana.plugins.[]. But we need to find a way to define and keep up to date the structure of the data that we return for every other fields (other stack_stats.[PRODUCT], and the stack_stats.kibana and root keys like version, cluster_stats, etc.).

This way we can have a contract with the remote telemetry service for all the fields that we send.

Items to cover:

• Define "static" definition of the schema
• Add functional tests that validate the payload against the schema to be aware of breaking changes in ES APIs (split definition between OSS and X-Pack)
• Check original issue where ES defines their contracts (Add aggregation list to node info elasticsearch#60074)
• ML specific use case: speak ES-ML devs to know how to best test this.
• Maybe train the devs that constantly change the contract? Include it in the wiki? Emails?

N.B.: Elasticsearch will eventually provide an API for telemetry purposes (elastic/elasticsearch#58198). But it's not happening anytime soon.

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[TinaHeiligers]

Additional tasks required to support this:

• Update the telemetry-tools
• Update the schema-to-mapping tools in the telemetry repo

Note: nodes/usage fields will need to be mostly dynamic. We might also need to get non-kibana teams to supply the schemas somehow.

[Bamieh]

I suggest that we keep away from complicating the tools we have to support the "static" part of the data we report. I think adding a static file for the remainder of the reported fields in the telemetry repo should be more than enough. I dont recall any changes to those fields since the past 2 years or so.

[afharo]

Makes sense! +1 on creating static schemas for that.

I'd add on top some sort of validation based on the schema we define in the API's functional tests to make sure we are compliant over time. How does that sound?

[joshdover]

+1 to including functional tests in the scope of this task

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

Updated the description with the agreed items to cover to in this issue.