[Security Solutions][POC] Create and integrate an OpenAPI specification for rules

[FrankHassanabad]

This is a experimental ticket/POC for uncovering any issues, problems if were to migrate to the OpenAPI specification: https://swagger.io/specification/

Other projects, tickets, discussions around OpenAPI specification:
#36575
#52284
ec-openapi-specification

Overview
As more and more people utilize rules across organizations, github repos, programming languages, data formats, etc... we are wanting to support solving problems where moving to a declarative and parseable specification such as OpenAPI would be helpful for everyone.

Our hope is to have a centralized definition for the most common sections of a rule when it exists on a file system, Kibana alerting Saved Object, http endpoint, a git repo, etc... and to be able to encode this within the OpenAPI specification. Other OpenAPI specifications would then include this specification such as our Kibana REST route request/response which would include put/post/patch/_import/_export specs, Kibana alerting storage specs, our pre-packaged rules file system spec, and future specs such as CDN based specs. OpenAPI's oneOf, anyOf, allOf, not look like they all would fit nicely for our needs and provide a path forward for unions based on alerting types with their discriminator feature.

Other repo's today outside of Kibana also have their own evolving rule specs which do data transforms into our existing rule spec and might benefit from being able to read from this specification for their own needs such as validations after a transform before an import.

Free things we get from OpenAPI we need

• Multi programming language support
• Docs generation (we also have internal doc tooling that supports this format already)
• Multi github repo and https schema references for other projects

Goals:

• Establish a directory structure/pattern in the detections engine for the schemas
• Create a starting single rule spec that can consumed via $ref for REST specs and other non-REST specs/validations
• Create REST request/response specs that use $ref and pull from this single rule spec
• Wire up REST validation and hook up existing unit tests to pass
• Determine TypeScript generation method and output folder via scripts if we generate them (might might/not be io-ts)
• Determine when using techniques such as discriminator if we get type unions or partials or etc...
• Reduce existing touch points when adding/modifying a rule field through this single spec and types
• Migrate existing docs to openAPI in the codebase and then re-wire doc generation tools to output from codebase
• Hopefully better performance and security
• Path for tooling teams to auto create client libraries from the specification
• Compare our approach to other teams and socialize pit falls, issues, etc...

Constraints:

• Don't break the existing REST API if we don't have to
• Types (at least) for both front and backend. Optionally having validation for front end would be good as an option but if we can't have it, it's not the end of the world.

Bridges still too far:

• Consuming Elastic declaration language since it's not in OpenAPI to get types from there. We utilize a few "pass through" REST type things directly to elastic or close to it such as filters on a query. Those can just stay as type "object" or "unknown"
• REST API versioning (but we are fine with schema versioning)
• Some types of data based validations that are not within the sweet spot of OpenAPI will be done right as the REST route accepts the message.
• Using a separate repo for this work similar to ECS. We are going to try to keep the directory structure local to Kibana for the short term and re-visit later.

Unknowns that should be checked off as we know them:

• Can we convert from snake_case to camelCase with the TypeScript generator? Likewise, are responses going to auto-convert from camelCase back to snake case? If not, then should we just embrace full on snake_case variables?
• PATCH might be problematic with OpenAPI through initial googling. If so, we might have to do a one-off for PATCH but don't want PATCH to be a large influencer of the schema model or cause large decomposition of keys and types within the schema as it has in the past.
• Are we more exposed through the generators to more security issues such as prototype pollution or less or about the same?
• Can we reduce our numbers of unit tests or auto generate tests and then "eye ball them"?
• Can we output Kibana alerting adapter validation using Kibana schema use case to reduce touch points or is that a bridge too far still?
• If using the discriminator feature are we getting strong union types between alert types?
• How do we remove the io-ts is guard usages of existing exception lists and other areas? Is that possible/easy?
• If we use generators for types do they all go into common folder or are they split up between common, public, server? Or can we use a derived type based system where it's "automatic" like typeOf?
• For the validator do we use this one? https://ajv.js.org/
• What is the performance with imports of large sets of rules? We had issues in the past.
• Can we get linter errors/warnings on people using camelCase vs. snake_case but add exceptions to existing problems we have?

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[legrego]

@FrankHassanabad I adjusted the team label: "Team:Security" is the other security team, not the team responsible for the Security Solution

[EricDavisX]

@jfsiii @ph @dedemorton interesting convo / ticket about API Docs and using OpenAPI spec... we've started doing that on the Fleet side, wasn't sure if anyone could / would want to sync up to see what both sides have done and can learn from etc.