[Security Solutions][Detection Engine] Meta - Migration of security_solution rules to Stack Management export/import/copy

[FrankHassanabad]

Describe the feature:

As an end user (Security Platform Engineer), I want to be able to use the Stack Management UI to:

• export
• import
• copy

security_solution rules and all its dependencies/references. As an end user (Security Platform Engineer) my responsibilities are for the set-up and maintenance of the SIEM/security analytics platform. Part of this role as an end user includes being able to export alerts and rules from one space to a separate version control system or air gapped separate system and then import them within the other system(s).

Another responsibility is the maintenance and addition/subtraction of Kibana spaces for different organizations depending on new organizations being created (for example). As an end user I would like the ability to copy and import rules between spaces as well, including the actions. As an end user I would like to do all of this from the Saved Object Management (SOM) system where I am already familiar with exporting/copying other data.

The export/copy should include everything that is a Saved Object (SO) along with references:

• Base rule definition
• Exception lists and exception list items
• Actions

The only exclusions to this would be:

• value lists

Currently value lists are data indexes. We will export shallow references to these indexes but until we migrate value lists to the saved object system this will not be included with the import/export. Rather users are expected to use other data index importing/exporting capabilities.

Ideally the export should be able to:

• Export all security_solution rules and their references and redact any sensitive information such as API keys
• Remove any non-needed data (but really we can live with extraneous data being exported)
• Look like the format of the other SO exports such as camelCase instead of snake_case.

Ideally the import should be able to:

• Adhere to security_solution existing business layer rules. These include things such as disallow immutable rules to be imported, querying existing rules by rule_id rather than SO id and overwriting those, bumping those version numbers if we import on top of them, etc...
• Import all the saved objects back into the same space and override the existing objects.
• Import the saved objects into a new space and all rules, connectors, exception lists should be imported as well.
• A user can first delete all their saved objects and then upon import, all the rules, connectors, exception lists, etc..., will come back into the system.
• There should not be duplicate ID's between the spaces such as list_id or other spots when we end up doing shared objects across spaces. This might include rule_id as another one.
• There should not be dangling or missing references when we do imports
• We can import our existing security_solution format and output a warning toaster.

Ideally the copy should be able to:

• Copy all the saved objects to a new space and override existing objects.
• Copy the saved objects into a new space and all rules, connectors, exception lists should be copied as well.
• A user can delete their saved objects from the first space and not experience data loss after copying.
• There should not be duplicate ID's between the spaces such as list_id or other spots when we end up doing shared objects across spaces. This might include rule_id as another one.
• There should not be dangling or missing references when we do copying
• Copying and importing should result in nearly if not the exact same data object that a "copy" would accomplish with regards to "created_by", "created_at", API keys, etc...

Below are issues created by security_solutions or other teams and are in priority order of what security_solution needs to be able to do export/import/copy of security_solution rules and its references. In some cases we can do workarounds or live with the consequences of when the work is not performed as technical debt and that will be listed below as well. I will avoid marking "blocked" on as much as possible but rather list consequences if something is not completed.

---

Import hook/interceptor (blocks us) nice to haves

[Update 2-1-22: Talking to product, we found these not to be blockers and more nice to haves. There are ways in which we could maintain validations client side if we find there lack there of in the SOM to be problematic. Our biggest blocker to moving to the SOM - security solution legacy actions - has been resolved.]

Issue:
This covers the data type validations, data shape validations, basic range validation written by the Kibana core team:
#104088

This covers the other aspects which is what we want for security_solutions to be able to update fields, add redacted fields, etc...
#109189

Similar issue ticket found here from @jonathan-buttner:
#110146

Use Cases:

As an end user (Security Platform Engineer), when I import rules I need the fields of updated_by, updated_at to be capable of being updated to the name of the engineer doing the import and the time it was updated.

As an end user (Security Platform Engineer) I need data validation to ensure that invalid combinations of rule parameters are not entered.

As an end user (Security Platform Engineer) I need UI feedback of any invalid rules, skipped rules, skipped exceptions, skipped exception list items, or unavoidable dangling issues due to data problems during the import.

As an end user (Security Platform Engineer), I would like to import the older export format from security_solution directly within this UI/UX screen and not have errors and it will be backwards compatible. It would be nice if I got a deprecated message that my older format will no longer work eventually and I need to re-do an export after an import to upgrade the format.

Technicals:
Right now within the Kibana core onImport, we cannot subtract invalid data objects, we cannot do data validations, and we cannot update fields such as updated_by, updated_at. This allows users to enter invalid data, cause unexpected UI breakages or other problems we want to avoid.

Also, we have an original import format for security_solution which is not the same as the Saved Object format. This format is based on a compact JSON format and we would like to retain backwards compatibility for a while if possible.

We also cannot use the rule_id if the SO id is not found to detect if an existing rule by a non SO id exists on the system. These are pre-packaged rules or if the user is importing offline rules created outside of the Kibana system. We might remove the rule_id but those ID's cannot be re-generated and is a feature of pre-pacakged rules.

What if we don't do this work?:
It is easy to break the user interfaces in a lot of ways with invalid data. Trouble shooting bad imports will be difficult and time consuming. The imports can have non-existent user names, or remove important data by accident. Users might use this as a feature and as a REST API to remove certain business rules we have making it harder to enforce the business rules such as an immutable rule flag, etc...

Users who import a set of rules from one space to another space will not see the updated_by, updated_at fields change when they should be changed.

You will get users with duplicate rule_id or overwriting the immutable rules or other business layer issues.

Add ability to know when a onCopy happens vs onExport

Issue:
#109190

Use Case:

As an end user (Security Platform Engineer), when I export vs. when I do a copy I need the Saved Object (SO) types of rules, exception lists, etc... to behave in two different ways.

1. The export functionality should preserve common fields such as updated_by, updated_at, etc...
2. The copying functionality should update these values to new values of last updated, who last updated it, etc.... The created_at, created_by should change to the author of who is copying it unless other examples show something different.

Technicals:
In the existing implementation, the Kibana core onExport is called for both copying and for exporting saved objects. The export option for rules already removes particular fields such as apiKey, apiKeyOwner, scheduledTaskId. Export also sets the value of enabled to be false.

However, on copying a rule and its references, we will need to re-provision the apiKey, apiKeyOwner and then update other fields such as updated_at, updated_by as well as do deep copying of exception lists so that the list_id and any other fields are not shallow linked between each other.

What if we don't do this work?:
Copying of data always keeps the same updated_by, created_by, updated_at fields when that is not really the last person to have touched it. The person who created the exception list or rules might not have access to the new space but the created_by will make people think that they did have access at some point when they might never have (confusion).

Data is redacted as if it is an export and we could run into a situation where the two still need to be separated and we would have to drop support for copying until this is fixed.

---

Other nice to have/referenced issues:

These are nice to have, referenced issues, but none of these are blocking us or impeding us on security_solution. If that changes, these will be moved up.

Issue:
#106581 (Better UX if user tries to delete hidden saved object)

---

- [Update 2-1-22: The below is still open and interesting to discuss, but no longer a consideration for us moving to SOM]

Add ability to query during saved object migrations

Issue:
#109188

Use Cases:
As an end user (Security Platform Engineer), when I copy or import the same exception list and its exception list items to two different spaces and then share both of the lists, they should show as completely unique.

As an end user (Security Platform Engineer), when I click on the relationships UI within the saved object management of an exception list I would expect the exception list to show all of the exception list items as children.

Technicals:
Right now within exception lists we do not store the exception list items as saved object references. Instead we opted to create an exception list id called list_id and then put that as a foreign key within exception list item documents. In order for us to migrate this so that the list_id is replaced with the exception list saved object id, we need to query for all of the children to put into the references array during migration when we encounter an exception list container.

Note that the exception list container and exception list items are all within the same saved object type to avoid blowing up the mappings of saved objects. We don't think this is going to be a problem if we have SO references to the same saved object types within the same saved object types.

Caveats are that the array for saved object references is going to potentially large (1k+), but possibly and realistically it could just end up being 10s of items and this might not ever be a problem. If it is, it will be when users are found to be using machine generated exception list items.

What if we don't do this work?:
You can end up with duplicate list_id keys whenever users use the import or copy functionality to import or copy exception lists between spaces. That is unless we add the features below where we can change out the list_id during a copy operation or during an import operation by querying the children list items and using the new generated reference id during those operations.

Unless there is an additional hook/interceptor for the references section of the Saved Object Management UI for gathering up visual references, the UI visual references section will always be empty in the UI.

Down the road we could encounter a different situation which dictates this still needs to be solved even if we add the other hooks to mitigate this.

We could do a different workaround outside of the saved object migrations somewhere in the UI or backend to hand fix this as well if we cannot get the feature from Kibana core.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[lukeelmers]

Heya @spong @FrankHassanabad -- After stringing together various slack threads and the newly created issues, I think I'm missing a higher level of context here, and am wondering if we could take a step back and ask a few (possibly naive) questions.

It feels a bit like we're jumping straight to solutions; it would help to have an explanation of the background and the problem space (the "how we got here" and "where we want to go" parts). I'm able to get bits and pieces from all the user stories above, but I'm still missing a clear picture of the current architecture and history.

Specifically, it'd be helpful to see:

• An outline of the existing security solutions architecture for handling imports/exports ("how we got here")
• The desired outcome / product requirements ("where we want to go")
  • Here it would be most helpful to focus on outcomes for the end-user, rather than for the platform developer. Some of this is already captured above, which is great 🙂
  • Core can help provide recommendations around how to solve this at a platform level once we fully understand the current vs desired state
• Which pieces of the experience (a) are not achievable with existing core services; and (b) cannot be worked around by security solutions.
  • This should just be a small subset of the items listed above
  • For these items, it is important to understand if they are "nice to haves" or "hard requirements"

While I expect some of the linked issues here will still be relevant, I don't think we have enough insight yet to quantify the actual impact this has on the core team's roadmap, understand how it stacks up against other organizational goals, and assess whether any of it is achievable in a 7.16 time frame.

We are mid-sprint now and another meeting to prioritize upcoming work in about 2 weeks; perhaps we can get some of these questions ironed out by then so the core team can formulate some recommendations for a path forward? Lmk what you think.

cc @thesmallestduck @alexfrancoeur @planadecu @alexh97

[spong]

Hey there @lukeelmers! 👋 Was out for a bit, so apologies for the delay here.

Hopefully I can provide some context to the needs from the security-solution side, and happy to meet to clear up any additional questions as well.

• An outline of the existing security solutions architecture for handling imports/exports ("how we got here")

So on the Detections side (can't speak for timelines, cases, etc), we started out with a custom import/export for Detection Rules when first releasing our beta back in 7.6 as it wasn't possible at the time to import/export Rule SO's backed by the Alerting framework. In subsequent releases we added support for Exception & Value Lists, both of which we ended up rolling our own import/export for as the necessary platform functionality wasn't available (no SO references[] on rules at the time, and value lists being an ES index). For an explanation of all the SO's Detections is/was managing, please see this comment. This probably sums up how we got here, with the main gist being we were often constrained by platform limitations and so found workarounds at the time in effort to provide near-term functionality to our users. A curated import/export experience also made for nice UX within the Security app.

• The desired outcome / product requirements ("where we want to go")
  • Here it would be most helpful to focus on outcomes for the end-user, rather than for the platform developer. Some of this is already captured above, which is great 🙂
  • Core can help provide recommendations around how to solve this at a platform level once we fully understand the current vs desired state

Fueled by the incoming breaking SO conversation work, a need to unite on stack-wide Rule management UX (i.e. RAC), and a desire to pay back the tech debt accrued by adding all this functionality, we're working towards ensuring all Detection entities have the appropriate SO references[] and are within spec such that they can be managed via the SOM UI. I think we'll want to keep our curated import/export UX within the Security app (defer to product here cc @jethr0null), but it seems best for us to work together with the existing stack infrastructure and just have custom UI for it vs maintaining all our own import/export logic. All that said, the most helpful outcome for our users is that they would have the ability to back up all their Detection entities with one button click and be able to restore them just as easily. This means a user should be able to export all their custom Rules, associated Exception/Value Lists (agnostic and space-aware), Actions, and Connectors, and then re-import them all either to another space or cluster, and just have to set the encrypted data on their Connectors and re-enable their rules to initialize their API keys.

• Which pieces of the experience (a) are not achievable with existing core services; and (b) cannot be worked around by security solutions.
  • This should just be a small subset of the items listed above
  • For these items, it is important to understand if they are "nice to haves" or "hard requirements"

At this moment (and correct my if I'm wrong here @FrankHassanabad), it's looking like we'll be able to get around most items except for Import hook/interceptor #109189, for the reasons mentioned above. As of now, we've been able to pay back a good bit of our tech debt and have been able to remove now-unnecessary side-car/shadow SO's like we had to implement for Rule Actions (#109722), and even have a draft PR up where we're able to export Rules + Exception Lists via the SOM UI (#109192) (this should also work with Connectors/Actions now as well too).

All that said, it might be best for us to just touch base next week and show you what we have working and how the above requests come into play, and then we can best prioritize from there. I'll reach out Monday and see if we can't get some time to chat. Thanks again for your attention here, really appreciate it! 🙂

[FrankHassanabad]

Here it would be most helpful to focus on outcomes for the end-user, rather than for the platform developer.

The "platform developer" is one of our end-users. We have different security engineer types for our Personas for our end-users and the "platform developer" is one of them. I changed the wording to try to make it more clear that it is from the end-user's perspective.

At this moment (and correct my if I'm wrong here @FrankHassanabad), it's looking like we'll be able to get around most items except for Import hook/interceptor #109189, for the reasons mentioned above.

Short term, we need at least the import one. Longer term we still need the other ones.