Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RAC][Rule Registry] Add unique ID field for identifying all alerts created during a specific Rule execution #110135

Closed
Tracked by #118324
spong opened this issue Aug 25, 2021 · 5 comments · Fixed by #113058
Closed
Tracked by #118324

[RAC][Rule Registry] Add unique ID field for identifying all alerts created during a specific Rule execution #110135

spong opened this issue Aug 25, 2021 · 5 comments · Fixed by #113058
Assignees
@spong
Labels
enhancement New value added to drive a business result Feature:EventLog Feature:Rule Monitoring Security Solution Detection Rule Monitoring Team:Detection Alerts Security Detection Alerts Area Team Team:Detection Rule Management Security Detection Rule Management Team Team:Observability Team label for Observability Team (for things that are handled across all of observability) Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: rac label obsolete v8.1.0

Comments

@spong
Copy link
Member

spong commented Aug 25, 2021 •
edited
Loading

This enhancement is for adding a new field to the global Alerts as Data schema for identifying all alerts created during a specific rule execution. This unique ID can also be logged out via the Rule Execution log to enable the ability to query for all alerts generated from that execution cycle. This is also useful for segmenting alerts within the UI when investigating further or debugging/tuning rules.

Currently, the closest field we have to this are the following two group fields:

kibana.alert.group.id

ID shared by all alerts within a group - currently used by EQL sequences to link the individual docs in the sequence together

kibana.alert.group.index

Index of the alert within the group (0, 1, etc). Used to track where the document was found in the sequence.

However they're currently used exclusively by EQL rules within Security. In our last schema sync, @marshallmain mentioned we could potentially move off of using these fields in favor of leveraging the evaluation event concept, so it's possible we could just go ahead and repurpose these fields for this enhancement (populating kibana.alert.group.id and removing kibana.alert.group.index). Alternatively, we could create a new field, e.g. kibana.alert.rule_execution_id.

At this time I don't think there's any need to encode any additional information in this ID, so should be fine to generate a UUID during each Rule execution and set it on all alerts (at a healthy abstraction away from specific rule types so that each rule writer doesn't need to implement this functionality, i.e. when getWriter() is first called).

If there's any useful information to encode here, or additional group sub-fields that may be useful, please do comment 🙂
@spong spong added enhancement New value added to drive a business result Team:Observability Team label for Observability Team (for things that are handled across all of observability) Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:EventLog Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: rac label obsolete labels Aug 25, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@spong spong mentioned this issue Aug 25, 2021
Merged
7 tasks
@banderror
Copy link
Contributor

++ for this enhancement 👍

I have a few random thoughts:

  • kibana.rule.execution.uuid - what do you think about this name? In the RFC for the Rule Execution Log we are proposing fields like kibana.rule.execution.status, and probably we could aim for consistency in the field names and also use the execution id field in all the contexts: Alerting's framework log, rule execution log, alerts-as-data.
  • Unique execution ID should probably be provided by the Alerting Framework rather than generated by our code.
  • This would allow us to use it as a correlation id between all sorts of logs and indices.

@spong
Copy link
Member Author

spong commented Aug 26, 2021 •
edited
Loading

kibana.rule.execution.uuid - what do you think about this name? In the RFC for the Rule Execution Log we are proposing fields like kibana.rule.execution.status, and probably we could aim for consistency in the field names and also use the execution id field in all the contexts: Alerting's framework log, rule execution log, alerts-as-data.

That sounds good to me! Note that within the AAD schema the rule namespace is nested under alert, so on that side of the house it'd probably be: kibana.alert.rule.execution.uuid (I've updated the spreadsheet accordingly).

Unique execution ID should probably be provided by the Alerting Framework rather than generated by our code.

All the better! 🙂 We'll still need to add it to the alert(s) being written within the RuleRegistry abstractions, but agree it makes most sense to have the alerting framework generate and inject this value.

@mikecote
Copy link
Contributor

It should be fairly easy to make the alerting framework provide such uuid (PRs welcome! 🙂). I think adding some code here is all that is necessary.

At the same time, if you want to add alert uuids to the framework (to align with alert as data ids), it can be done in this file. Probably somewhere in the constructor and toRaw, I'm guessing.

@spong spong self-assigned this Sep 10, 2021
@spong spong added Feature:Rule Monitoring Security Solution Detection Rule Monitoring v7.16.0 labels Sep 10, 2021
@peluja1012 peluja1012 added Team:Detection Rule Management Security Detection Rule Management Team Team:Detection Alerts Security Detection Alerts Area Team labels Sep 15, 2021
This was referenced Sep 24, 2021
Merged
Merged
@afgomez afgomez mentioned this issue Oct 12, 2021
Closed
@spong spong added v8.0.0 and removed v7.16.0 labels Nov 4, 2021
@banderror banderror mentioned this issue Nov 11, 2021
Open
19 tasks
spong added a commit that referenced this issue Jan 21, 2022
@spong
[RAC][Alerting][Security Solution] Adds Rule Execution UUID (#113058)
48693d1 
## Summary

Resolves: #110135

This PR is for introducing a new UUID (`kibana.alert.rule.execution.uuid` as defined in the AAD schema) for identifying individual rule executions. This id is introduced as a `private readonly` member of the [alerting server task_manager](https:/elastic/kibana/blob/a993668663dd4fc25d3336e2d474101ed8d1b74d/x-pack/plugins/alerting/server/task_runner/task_runner.ts#L123), and plumbed through the `executionHandler` and to all appropriate alert event and event-log touch points.

For persistence when writing alerts within the RuleRegistry, `kibana.alert.rule.execution.uuid` is plumbed through [`getCommonAlertFields()`](https:/elastic/kibana/blob/c81341c325edcb0eaca1dab2521b2a86fea18389/x-pack/plugins/rule_registry/server/utils/get_common_alert_fields.ts#L52) so it is grouped with like fields and is picked up by both the [`createPersistenceRuleTypeWrapper`](https:/elastic/kibana/blob/c81341c325edcb0eaca1dab2521b2a86fea18389/x-pack/plugins/rule_registry/server/utils/create_persistence_rule_type_wrapper.ts#L38) used by Security Solution, and [`createLifecycleExecutor`](https:/elastic/kibana/blob/d152ca5b6bf7f56fcba1d1d8c2cfee5404a821de/x-pack/plugins/rule_registry/server/utils/create_lifecycle_executor.ts#L157) used by Observability rules.

Additionally on the Security Solution side, `kibana.alert.rule.execution.uuid` was plumbed through the `RuleExecutionLog` so that all events written to the event-log will now include this id so individual rule status events/metrics can be correlated with specific rule executions.

No UI facing changes were made, however `kibana.alert.rule.execution.uuid` is now available within the Alerts Table FieldBrowser, and can be toggled and viewed alongside alerts:

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/149594433-b16e369b-adf4-4ae3-b6e9-98189b214d51.png" />
</p>

As visible when exploring `event-log` in Discover:

<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/149595237-155d25d7-8324-4981-9ec2-faf0af8e7bd7.png" />
</p>



### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials 
  - Will need to sync with Doc folks on updates here.
- [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@kobelb kobelb added the needs-team Issues missing a team label label Jan 31, 2022
@botelastic botelastic bot removed the needs-team Issues missing a team label label Jan 31, 2022
@banderror banderror mentioned this issue Feb 8, 2022
Open
@benironside benironside mentioned this issue Feb 28, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spong spong

Labels
enhancement New value added to drive a business result Feature:EventLog Feature:Rule Monitoring Security Solution Detection Rule Monitoring Team:Detection Alerts Security Detection Alerts Area Team Team:Detection Rule Management Security Detection Rule Management Team Team:Observability Team label for Observability Team (for things that are handled across all of observability) Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: rac label obsolete v8.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[RAC][Alerting][Security Solution] Adds Rule Execution UUID spong/kibana
6 participants
@peluja1012 @kobelb @spong @mikecote @banderror @elasticmachine