[Security Solutions][Detection Engine][Meta][Remix] - Migration of security_solution rules to Stack Management export/import/copy

[yctercero]

Objective

At it's most basic, we want to enable security solution rules to be exported using the Stack Management UI/APIs and begin deprecating our own import/export API.

A ticket that dives more into the technical details and different considerations/use cases thought through exists here. This issue is meant to be more high level.

Security Solution Rules At a Glance

Current Experience

As a security solution user, I want to export my security solution rules and all saved objects associated with it:

• Go to the security solution rule management UI and select rules to export
• Export file includes rules and exceptions
• Connectors are not included, you must export those by going to the Stack Management UI

As a user of multiple solutions, I want to export my rules and all saved objects associated with it:

• Go to the stack management page and import/export your non security solution rules
• Go to the security solution rule management UI and select rules to export
• Export file includes rules and exceptions
• Connectors are not included, you must export those by going to the Stack Management UI
• Rules from SOM and rules exported from security solution will be exported in two different formats

Proposed Experience

As a security solution user, I want to export my security solution rules and all saved objects associated with it:

• Go to the rules management page and import/export is available in the same format as the SOM
• OR go to the stack management page and import/export rules

As a user of multiple solutions, I want to export my rules and all saved objects associated with it:

• Go to the stack management page and import/export rules (not needing to think through which solution the rules belong to)

Exclusions

• Value lists (stored in data indices)
  • Not currently accessible within our own import/export - no value added/subtracted here in moving to the SOM

Possible effects to user experience

• No support for data validation on import/export. More details discussed here.
  • -Our current security solution import/export endpoints conduct thorough validations
  • -This means users can break themselves by importing an invalid structure
  • +However, they are provided a way to fix themselves and additional validation can be added client side (on our end, not talking about the SOM client)
  • +Other solutions currently use the SOM as is

TLDR

This effort is more than just a matter of user experience. On the technical side, security solution is currently maintaining a separate export/import API and needing to implement functionality to maintain parity with the SOM. Complexity has increased as shareable saved objects are introduced.

Open questions

• Security solution makes use of signature ids (rule_id, timeline_template_id, item_id, list_id) - is there any support for this in core? What would signature id clashes look like in export/import?
• Can timeline move to SOM, if not, can we exclude? What's needed to move over?
• Can trusted apps move to SOM, if not, can we exclude? What's needed to move over?

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[jethr0null]

Huge +1 on the proposed experience.

[yctercero]

NOTE: It appears that meta fields in the SOM are taken from the import file and written to the SOs (created_by, created_at, udpated_by, updated_at). This diverges from our current import/export that prevented a user from specifying these fields for security purposes.

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)