[Security Solution] [Sourcerer] [Feature Branch] Update to use Kibana Data Views

[stephmilovic]

Summary

Closing #111461 in favor of this PR to create a "feature branch", they are essentially the same PR

Kibana Index Patterns Data Views & Runtime Fields in Security Solution Sourcerer

Issue: https:/elastic/security-team/issues/772

• Removes config index patterns and only use Kibana index patterns. Create a special security-solution KIP from the config index patterns. PR Reviewed: [Security Solution] [Sourcerer] KIP Feature Branch Kickoff, remove config index patterns #106460
• Changes to the config index patterns to update the security-solution Kibana data view
• Add signals index to default Kibana data view
• Updates requestIndexFieldSearch in timelines to take a dataViewId
• Switch to get fields directly from Kibana data view
• Enable sourcerer to break up indices on each Kibana data view once selected
• Update timeline sourcerer ui (pick_events.tsx)
• Add dataViewId property to timeline, stored in references on the saved object

Quirks

• data view titles allow for duplicate indices, so we need to allow for duplicates to display accurate data view titles. For example: auditbeat-*,auditbeat-*,auditbeat-*,auditbeat-* and auditbeat-*,auditbeat-* are both valid. We're only going to allow them to select one instance of auditbeat from our combo box, but the select needs the full name so the user can distinguish between the data views and know where they put their runtime fields
  • there is another spot where i know you'll be like, why check for duplicates here? its in case something wonky is in the url state
• the data view gets updated when the Advanced Settings default index is updated securitySolution:defaultIndex. A page refresh is required for updates to take place
• the siem signals index is appended to the end of the security-solution data view. on initial app load, or if the signals index gets deleted, signalIndexName is null and quickly set. We have a call to update the security-solution data view when the signalIndexName is updated
• when the data view gets deleted and you navigate back to the security solution, a page refresh will be forced to regenerate the data view
• to check if an index pattern is valid, we are now using esClient.fieldCaps instead of esClient.search or esClient.count as both of those methods require there to be data on the index pattern. When .siem-signals-default is initially created, it will not immediately have data. So using esClient.fieldCaps will tell us if the index exists even without data

UI Updates coming in a follow up PR. See Monina's excellent design direction in the original issue: https:/elastic/security-team/issues/772

Sourcerer Updates to KIP/Data Views:

Runtime Fields in Security Solution:

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[angorayc]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 6a68772
• Storybooks Preview

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
securitySolution	1318	1328	+10
timelines	848	861	+13
total			+23

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
observability	375.7KB	375.7KB	+19.0B
securitySolution	4.5MB	4.5MB	+6.8KB
timelines	236.0KB	236.2KB	+194.0B
total			+7.0KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	246.6KB	248.9KB	+2.2KB
timelines	156.8KB	157.0KB	+147.0B
total			+2.4KB

Unknown metric groups

API count

id	before	after	diff
securitySolution	1372	1377	+5
timelines	969	977	+8
total			+13

References to deprecated APIs

id	before	after	diff
securitySolution	1012	542	-470
timelines	84	52	-32
total			-502

History

• 💔 Build #4167 failed 8286a33
• 💔 Build #3476 failed 7b13681
• 💔 Build #3148 failed 2c8adfd
• 💔 Build #2819 failed 479014c
• 💔 Build #2720 failed 3704a28

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream