[8.0 only] Make rule and connector saved objects share-capable

[mikecote]

Problem Statement

In order to support sharing rules and connectors in a minor release (at some point in 8.x), we need to make some breaking changes in advance (in 8.0) to make sure our saved objects "ready" for sharing at a future time. We did not complete the research yet to know the exact changes that are necessary to our codebase for the readiness to be successful. Let's use this issue to research the necessary changes, work with the Kibana Security team and use #100489 to identify what changes are necessary for 8.0.

---

As part of the sharing saved-objects in multiple spaces project, we need to convert all of our space-specific saved object types to share-capable / multiple-isolated namespace type in the 8.0 release (meta: #100489). This is required to enable us at a future time to support sharing our saved objects across spaces.

After talking with @jportner, this 8.0 migration will remove the namespace from the serialized saved object ID (ex: space-1:rule:123 -> rule:123), and generate a new object ID, which is why this needs to happen in a major version as a breaking change. For this to work, we'll need to make sure any place storing our SO IDs uses saved object references or is considered part of our migration.

Things top of mind:

• The case SO needs to use references
• When using references, only the reference array should store the ID of another object.
• We'll need to research how this works for pre-packaged rules/support for custom IDs Research approach for pre-defined rule IDs when making rules share-capable #100667
• We'll need to research how this works for the event log, alert data indices, etc Research approach for event log when making rules and connectors share-capable #100666

A lot of details are missing in this issue on how to resolve this. More details will come as the Kibana security team finalizes the work. It is still good to research in advance to see if we need anything further based on my top of mind list.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

cc @legrego

[ymao1]

The alerting framework is responsible for the action, alert and action_task_params saved object types. When trying to make these types share-capable, we need to look at:

• other saved objects that reference our saved object types
• rules and connectors that reference other saved object types in their rule params

Other saved objects that reference action, alert and action_task_params SO types:

1. cases - the cases SO stores the associated action SO ID in the cases.connector.id field and does not utilize the references array.

Example `cases` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases:554e2720-e8b2-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases": {
			"type": "individual",
			"title": "test case",
			"tags": [],
			"description": "hi",
			"connector": {
				"id": "7b0cf000-e8a7-11eb-bc58-13b6ae6d011a",
				"name": "test servicenow",
				"type": ".servicenow",
				"fields": [{
						"key": "urgency",
						"value": null
					},
					{
						"key": "severity",
						"value": null
					},
					{
						"key": "impact",
						"value": null
					},
					{
						"key": "category",
						"value": null
					},
					{
						"key": "subcategory",
						"value": null
					}
				]
			},
			"settings": {
				"syncAlerts": true
			},
			"owner": "securitySolution",
			"closed_at": null,
			"closed_by": null,
			"created_at": "2021-07-19T16:56:59.796Z",
			"created_by": {
				"email": null,
				"full_name": null,
				"username": "elastic"
			},
			"external_service": null,
			"status": "open",
			"updated_at": "2021-07-19T17:04:19.545Z",
			"updated_by": {
				"full_name": null,
				"email": null,
				"username": "elastic"
			}
		},
		"type": "cases",
		"references": [],
		"migrationVersion": {
			"cases": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:04:19.548Z"
	}
}

2. cases-user-actions - the cases-user-actions SO stores the associated action SO ID when the user changes the connector associated with a case. It does not use the references array and furthermore, the old and new action SO IDs are stored as part of a stringified JSON.

Example `cases-user-actions` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases-user-actions:99836640-e8a7-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases-user-actions": {
			"action_field": [
				"connector"
			],
			"action": "update",
			"action_at": "2021-07-19T15:40:09.374Z",
			"action_by": {
				"email": null,
				"full_name": null,
				"username": "elastic"
			},
			"new_value": """{"id ":"7b0cf000- e8a7- 11eb-bc58-13b6ae6d011a ","name ":"test servicenow ","type ":".servicenow ","fields ":null}""",
			"old_value": """{"id ":"6e67ec46-28af-4c19-8f19-683d6223ae26","name ":"service now personal developer instance ","type ":".servicenow ","fields ":{"urgency ":null,"severity ":null,"impact ":null,"category ":null,"subcategory ":null}}""",
			"owner": "securitySolution"
		},
		"type": "cases-user-actions",
		"references": [{
			"type": "cases",
			"name": "associated-cases",
			"id": "270e4080-e8a7-11eb-bc58-13b6ae6d011a"
		}],
		"migrationVersion": {
			"cases-user-actions": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T15:40:09.764Z"
	}
}

3. cases-connector-mappings - the cases-connector-mappings SO stores the associated action SO ID as part of its references array.

Example `cases-connector-mappings` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases-connector-mappings:561b1410-e8b2-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases-connector-mappings": {
			"mappings": [{
					"source": "title",
					"target": "short_description",
					"action_type": "overwrite"
				},
				{
					"source": "description",
					"target": "description",
					"action_type": "overwrite"
				},
				{
					"source": "comments",
					"target": "work_notes",
					"action_type": "append"
				}
			],
			"owner": "securitySolution"
		},
		"type": "cases-connector-mappings",
		"references": [{
			"type": "action",
			"name": "associated-action",
			"id": "7b0cf000-e8a7-11eb-bc58-13b6ae6d011a"
		}],
		"migrationVersion": {
			"cases-connector-mappings": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T16:57:01.137Z"
	}
}

4. cases-configure - the cases-configure SO stores the associated action SO ID in the connector.id field and does not utilize the references array

Example `cases-configure` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "cases-configure:2d5dbcc0-e8b3-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"cases-configure": {
			"connector": {
				"id": "7b0cf000-e8a7-11eb-bc58-13b6ae6d011a",
				"name": "test servicenow",
				"type": ".servicenow",
				"fields": []
			},
			"closure_type": "close-by-user",
			"owner": "securitySolution",
			"created_at": "2021-07-19T17:03:02.286Z",
			"created_by": {
				"username": "elastic",
				"email": null,
				"full_name": null
			},
			"updated_at": null,
			"updated_by": null
		},
		"type": "cases-configure",
		"references": [],
		"migrationVersion": {
			"cases-configure": "7.14.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:03:02.571Z"
	}
}

5. siem-detection-engine-rule-status - the siem-detection-engine-rule-status SO stores the associated alert SO ID in the alertId field and does not utilize the references array

Example `siem-detection-engine-rule-status` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "siem-detection-engine-rule-status:ab8f4fc0-e8b6-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"siem-detection-engine-rule-status": {
			"alertId": "a9f99490-e8b6-11eb-bc58-13b6ae6d011a",
			"statusDate": "2021-07-19T17:34:38.755Z",
			"status": "partial failure",
			"lastFailureAt": null,
			"lastSuccessAt": "2021-07-19T17:34:38.755Z",
			"lastFailureMessage": null,
			"lastSuccessMessage": """The following indices are missing the timestamp field "@timestamp ": [".kibana_8 .0 .0_001 "]""",
			"gap": null,
			"bulkCreateTimeDurations": [],
			"searchAfterTimeDurations": [],
			"lastLookBackDate": null
		},
		"type": "siem-detection-engine-rule-status",
		"references": [],
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:34:38.757Z"
	}
}

6. siem-detection-engine-rule-actions the siem-detection-engine-rule-actions SO stores the associated alert SO ID in the ruleAlertId field and associated action SO IDs in the actions array and does not utilize the references array

Example `siem-detection-engine-rule-actions` saved object

{
	"_index": ".kibana_8.0.0_001",
	"_id": "siem-detection-engine-rule-actions:ab321490-e8b6-11eb-bc58-13b6ae6d011a",
	"_score": 0.0,
	"_source": {
		"siem-detection-engine-rule-actions": {
			"ruleAlertId": "a9f99490-e8b6-11eb-bc58-13b6ae6d011a",
			"actions": [{
				"action_type_id": ".email",
				"id": "8fd8e910-e8a4-11eb-bc58-13b6ae6d011a",
				"params": {
					"subject": "hi",
					"to": [
						"[email protected]"
					],
					"message": "Rule {{context.rule.name}} generated {{state.signals_count}} alerts"
				},
				"group": "default"
			}],
			"ruleThrottle": "rule",
			"alertThrottle": null
		},
		"type": "siem-detection-engine-rule-actions",
		"references": [],
		"migrationVersion": {
			"siem-detection-engine-rule-actions": "7.11.2"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-19T17:37:28.027Z"
	}
}

Rule and connector types that reference other saved object types in their rule params

1. Security solution rule types - x-pack/plugins/security_solution/server/plugin.ts - these rules stores any associated exception-list SO IDs inside the rule params exceptionsList field. It also stores a params.ruleId that is NOT the associated alert SO ID and uses that inside the rule tags as well. Need to determine if that's something we need to worry about. Also, inside the exceptionList array, there is an id and a list_id. The id field looks to correspond to an exception-list SO, but have not figured out what the list_id corresponds to.

Example security solution rule

{
	"_index": ".kibana_8.0.0_001",
	"_id": "alert:202af230-e96b-11eb-a858-c5aee1479d55",
	"_score": 0.0,
	"_source": {
		"alert": {
			"name": "test",
			"tags": [
				"__internal_rule_id:44569889-d223-49c3-b33d-2bd0527f5ee6",
				"__internal_immutable:false"
			],
			"alertTypeId": "siem.signals",
			"consumer": "siem",
			"params": {
				"author": [],
				"description": "test",
				"falsePositives": [],
				"from": "now-75s",
				"license": "",
				"outputIndex": ".siem-signals-default",
				"meta": {
					"from": "1m",
					"kibana_siem_app_url": "https://localhost:5601/app/security"
				},
				"maxSignals": 100,
				"riskScore": 21,
				"riskScoreMapping": [],
				"severity": "low",
				"severityMapping": [],
				"threat": [],
				"to": "now",
				"references": [],
				"version": 2,
				"exceptionsList": [{
					"id": "3e0ec560-e96b-11eb-a858-c5aee1479d55",
					"list_id": "4fc01463-730f-4623-9ab4-09b762aa8469",
					"type": "detection",
					"namespace_type": "single"
				}],
				"ruleId": "44569889-d223-49c3-b33d-2bd0527f5ee6",
				"immutable": false,
				"query": "system.cpu.total.norm.pct > 0.2",
				"language": "kuery",
				"filters": [],
				"index": [
					"apm-*-transaction*",
					"traces-apm*",
					"auditbeat-*",
					"endgame-*",
					"filebeat-*",
					"logs-*",
					"packetbeat-*",
					"winlogbeat-*",
					"es-apm*"
				],
				"type": "query"
			},
			"schedule": {
				"interval": "15s"
			},
			"enabled": true,
			"actions": [],
			"throttle": null,
			"notifyWhen": "onActiveAlert",
			"apiKeyOwner": "elastic",
			"apiKey": "hV5CssnHuvmu+BPdK1nd33n68tD1ZJBJy/ipdxLpJxKho9Tmx3Foc8BRquyxd55tjLRdd9hG6b060v86IJGwW17p8yUf9QFnSIQqF4gsIv1IymUjFHc+eP/t5wBho3nJyqSLTQpX2zhiAHp+s/y5rT3wH2EeHmpAdeHHK/+BuaHunptwuisaypSXW6eEie3ww0LNANeoH1Jn/w==",
			"createdBy": "elastic",
			"updatedBy": "elastic",
			"createdAt": "2021-07-20T14:59:48.591Z",
			"updatedAt": "2021-07-20T15:00:39.717Z",
			"muteAll": false,
			"mutedInstanceIds": [],
			"executionStatus": {
				"status": "ok",
				"lastExecutionDate": "2021-07-20T15:41:17.191Z",
				"error": null
			},
			"meta": {
				"versionApiKeyLastmodified": "8.0.0"
			},
			"scheduledTaskId": "2160b310-e96b-11eb-a858-c5aee1479d55"
		},
		"type": "alert",
		"references": [],
		"migrationVersion": {
			"alert": "7.13.0"
		},
		"coreMigrationVersion": "8.0.0",
		"updated_at": "2021-07-20T15:41:18.965Z"
	}
}

2. Maps geo containment rule type - x-pack/plugins/stack_alerts/server/alert_types/geo_containment/index.ts - maps rules contain two index-pattern SO IDs inside its rule parameters

Example geo containment rule

{
	"alert": {
		"params": {
			"index": "tracks*",
			"indexId": "f3acd150-e8c6-11eb-bc58-13b6ae6d011a",
			"entity": "azimuth",
			"dateField": "@timestamp",
			"boundaryType": "entireIndex",
			"geoField": "location",
			"boundaryIndexTitle": "manhattan_boundaries",
			"boundaryIndexId": "a2e7a1f0-e8c6-11eb-bc58-13b6ae6d011a",
			"boundaryGeoField": "coordinates",
			"boundaryNameField": "<nothing selected>"
		},
		"consumer": "alerts",
		"schedule": {
			"interval": "1m"
		},
		"tags": [],
		"name": "map",
		"actions": [],
		"enabled": true,
		"throttle": null,
		"alertTypeId": ".geo-containment",
		"notifyWhen": "onActionGroupChange",
		"apiKeyOwner": "elastic",
		"apiKey": "HcRdCvA7pjzG170nbE5NJdtSPB2fQ/yKy3i+rOApxnjCw/G/7MaA5X1sSblceYMD5u7eoot3CKiFFOgKeiz3w9Glmvnk4scpGLAtHdQYn4lUtY5m5lYaO5Xzj+v8SIN/Ogqpx8/Tg+qLSVPRlpEWNrBh+sPnL0k0SofOfUkIurU4TRJQq4yDjZIf+DI6hgA1qOFd1aQckgnq0A==",
		"createdBy": "elastic",
		"updatedBy": "elastic",
		"createdAt": "2021-07-19T19:25:15.787Z",
		"updatedAt": "2021-07-19T19:25:15.787Z",
		"muteAll": false,
		"mutedInstanceIds": [],
		"executionStatus": {
			"status": "active",
			"lastExecutionDate": "2021-07-19T19:26:22.339Z",
			"error": null
		},
		"meta": {
			"versionApiKeyLastmodified": "8.0.0"
		},
		"scheduledTaskId": "0c1068b0-e8c7-11eb-bc58-13b6ae6d011a"
	},
	"type": "alert",
	"references": [],
	"namespace": "my-space",
	"migrationVersion": {
		"alert": "7.13.0"
	},
	"coreMigrationVersion": "8.0.0",
	"updated_at": "2021-07-19T19:26:22.631Z"
}

Note that the ML anomaly detection rule type references a ml-job type SO but that is namespaceType: 'multiple'.

[chrisronline]

I started doing some basic ad-hoc testing around this by creating multiple spaces and a rule/connector in each space on master. I then changed the above saved object types to the "share-capable" namespaceType: 'multiple-isolated' and restarted Kibana.

This is what happened:

server    log   [11:19:24.253] [warning][alerting][plugins] Unable to execute rule "e94d7740-e96c-11eb-b5b6-dd32b9d43f52" in the "chrisspace" space because Saved object [alert/e94d7740-e96c-11eb-b5b6-dd32b9d43f52] not found - this rule will not be rescheduled. To restart rule execution, try disabling and re-enabling this rule.
server    log   [11:19:24.254] [error][plugins][taskManager] Task alerting:.es-query "ea861e50-e96c-11eb-b5b6-dd32b9d43f52" failed: Error: Saved object [alert/e94d7740-e96c-11eb-b5b6-dd32b9d43f52] not found
server    log   [11:19:24.255] [warning][alerting][plugins] Unable to execute rule "c0a28bf0-e96c-11eb-b5b6-dd32b9d43f52" in the "robersonspace" space because Saved object [alert/c0a28bf0-e96c-11eb-b5b6-dd32b9d43f52] not found - this rule will not be rescheduled. To restart rule execution, try disabling and re-enabling this rule.
server    log   [11:19:24.256] [error][plugins][taskManager] Task alerting:.es-query "c18a53e0-e96c-11eb-b5b6-dd32b9d43f52" failed: Error: Saved object [alert/c0a28bf0-e96c-11eb-b5b6-dd32b9d43f52] not found

The UI doesn't show any problem though. That's definitely something we'll need to address (which I'm sure we knew about before I did this)

[chrisronline]

Very early research exposed a potential bug on the core/security side: #106567

@jportner has a fix for the above (#106897) which will make it a non issue!

[ymao1]

Created the following issues for our external dependencies:

• Maps rule type should use references array for index pattern references - [Maps][Alerting] Migrate to using Saved Object References array to reference index patterns from Rules #107066
• Cases should use references array for action references - [Cases][Connectors] Migrate to using Saved Object References array to reference action saved objects from Cases #107067
• Security should use references array for alert references - [Security Solution][Detections] Migrate to using Saved Object References array to reference alert saved objects from security saved objects #107068

[chrisronline]

Created the following issues for refactoring .get() to .resolve():

• Actions [Actions] [7.x] Convert all savedObjectClient.get() calls to use savedObjectClient.resolve and handle the response appropriately #107069
• Alerting [Alerting] [7.x] Convert all savedObjectClient.get() calls to use savedObjectClient.resolve and handle the response appropriately #107070
• Cases [Cases] Convert all savedObjectClient.get() calls to use savedObjectClient.resolve and handle the response appropriately #107071
• Encrypted saved objects [ESO] Convert all savedObjectClient.get() calls to use savedObjectClient.resolve and handle the response appropriately #107072

[ymao1]

Follow-up issues created so closing this research issue and moving to Done.

[chrisronline]

I'm reopening this as the original research needs to be adjusted based on some recent conversations between our team and the security team.

[chrisronline]

Based on the most recent round of research, this are the work items:

• [Alerting][7.x] Update rules detail page to use new saved object resolve API when necessary #108074
• [Alerting][7.x] Add migration that copies existing rule saved object ids to another location in the saved object #108075
• [Event log][7.x] Update event log client to search across legacy IDs #108077
• [Actions][7.x] Move saved object fields to references for action_task_params #108082
• [Task Manager] [8.0] Add migrations for "actions:* and "alerting:*" task types #108084
• [Alerting] [8.0] Prepare for making alerting saved objects sharecapable #107084
• [Actions] [8.0] Prepare for making action saved objects sharecapable #107083

[jportner]

In case anyone is following this issue but not the linked meta-issue (#100489):

I published the Sharing Saved Objects developer guide, please take a look before proceeding with the implementation. And of course, please reach out if you run into any problems!