elastic · stephmilovic · Apr 19, 2022 · Apr 6, 2022 · Apr 6, 2022 · Apr 11, 2022
@@ -0,0 +1,53 @@
+# Sourcerer Component
+
+
+### Adding sourcerer to a new page
+- In order for the sourcerer to show up on a page, it needs to be added to the array `sourcererPaths` in `containers/sourcerer/index.tsx`
+- The scope of a sourcerer component will be default unless the path is added to the `detectionsPaths` array, in which case the scope can be detections in `containers/sourcerer/index.tsx`
+
+## Default Sourcerer
+![](../../images/default.png)
+- The data view is shown in the dropdown. All index aliases that have existing indices are selected by default. Index aliases that do NOT have existing indices are shown in the dropdown as deactivated
+
+## Detections Sourcerer
+![](../../images/alerts.png)
+- The detections sourcerer is locked to the signals index, and has an "Alerts" flag
+
+## Timeline Sourcerer
+![](../../images/timeline.png)
+- The timeline sourcerer is almost exactly like the default sourcerer, but it has a checkbox to "Show only detection alerts" which will automatically select the default data view with the alerts index locked in:
+![](../../images/timeline-alerts.png)
+
+## Sourcerer Quirks
+ - Signals index is not always defined yet, so we `pollForSignalIndex` on mount of detections sourcerer and timeline sourcerer
+ - Signals index is hidden in default scope as it is never used on our Explore pages where default scope is used
+ - When `selectedPatterns` changes, we updated the `onUpdateDetectionAlertsChecked` boolean
+ - Reset button sets to the active `patternList` on the default security data view
+ - The "Modified" label gets applied when the data view `selectedPatterns` diverge from the `patternList`. It does not appear when changing data views and using that data views patternList in full. It does not appear until Save is pressed!
+ ![](../../images/timeline_alerts.png)
+
+## Legacy Timeline Sourcerer
+ - "Update available" workflow for pre-8.0 timelines and timelines that have indices that got deleted from the default data view
+
+### Flow 1
+**Note:** i paused the most in this flow so y'all can ready the copy, the rest of the flows go quicker
+Legacy Timeline includes an active index pattern that is not included in the default data view, user decides to update data view. Page refresh is prompted
+![f1](https://user-images.githubusercontent.com/6935300/144764137-d53b2468-db42-4031-81aa-716b4dfbfa89.gif)
+### Flow 2
+Legacy Timeline includes an active index pattern that is not included in the default data view, user decides to update to the new sourcerer with only the indices that already exist in the data view
+![f2](https://user-images.githubusercontent.com/6935300/144764142-f0311e7c-1a01-4258-a607-7e446a6649a9.gif)
+### Flow 3
+Legacy Timeline includes an active index pattern that is not included in the default data view, user decides to reset to the new sourcerer and abandon their index pattern
+![f3](https://user-images.githubusercontent.com/6935300/144764143-48a8d779-c413-4622-9929-54f544057fc4.gif)
+### Flow 4
+Legacy Timeline index patterns are ALL included in the default data view, user updates to the new sourcerer without needing to update the data view
+![f4](https://user-images.githubusercontent.com/6935300/144764144-005933ce-134b-45ec-b437-491b6c9a8bbf.gif)
+### Flow 5
+Legacy Timeline none of the index patterns in the legacy timeline match, don't allow user to upgrade data view. Forces them to reset to new sourcerer or keep a bad index pattern
+![f5](https://user-images.githubusercontent.com/6935300/144764145-7b079933-0847-4090-ab94-eada377e3e9a.gif)
+### Flow 6
+Start with a valid non-legacy timeline. Delete one of the index patterns from advanced settings. Fallsback into temporary timeline and prompts user to re-add the deleted index pattern
+![f6](https://user-images.githubusercontent.com/6935300/144764146-6e0ac904-4a8e-4a4c-bf4f-c43db07e5a79.gif)
+### Flow 7
+Error state, not expected
+![f7](https://user-images.githubusercontent.com/6935300/144874843-ff3e5d1a-3436-41f6-97af-cad3431b83cc.gif)
@@ -9,7 +9,7 @@ import React, { useCallback, useEffect, useMemo, useState, useRef } from 'react'
 import { EuiComboBoxOptionOption, EuiSuperSelectOption } from '@elastic/eui';
 import { useDispatch } from 'react-redux';
 
-import { getSourcererDataview } from '../../containers/sourcerer/api';
+import { getSourcererDataView } from '../../containers/sourcerer/api';
 import { getScopePatternListSelection } from '../../store/sourcerer/helpers';
 import { sourcererActions, sourcererModel } from '../../store/sourcerer';
 import { getDataViewSelectOptions, getPatternListWithoutSignals } from './helpers';
@@ -192,7 +192,7 @@ export const usePickIndexPatterns = ({
  setSelectedOptions([]);
  // TODO We will need to figure out how to pass an abortController, but as right now this hook is
  // constantly getting destroy and re-init
- const pickedDataViewData = await getSourcererDataview(newSelectedDataViewId);
+ const pickedDataViewData = await getSourcererDataView(newSelectedDataViewId);
  if (isHookAlive.current) {
  dispatch(
  sourcererActions.updateSourcererDataViews({

@@ -24,7 +24,7 @@ import { useAppToasts } from '../../hooks/use_app_toasts';
 import { sourcererActions } from '../../store/sourcerer';
 import * as i18n from './translations';
 import { SourcererScopeName } from '../../store/sourcerer/model';
-import { getSourcererDataview } from '../sourcerer/api';
+import { getSourcererDataView } from '../sourcerer/api';
 
 export type IndexFieldSearch = (param: {
  dataViewId: string;
@@ -121,7 +121,7 @@ export const useDataView = (): {
  };
  setLoading({ id: dataViewId, loading: true });
  if (needToBeInit) {
- const dataViewToUpdate = await getSourcererDataview(
+ const dataViewToUpdate = await getSourcererDataView(
  dataViewId,
  abortCtrl.current[dataViewId].signal
  );

@@ -31,7 +31,7 @@ export const postSourcererDataView = async ({
  signal,
  });
 
-export const getSourcererDataview = async (
+export const getSourcererDataView = async (
  dataViewId: string,
  signal?: AbortSignal
 ): Promise<KibanaDataView> => {

@@ -394,11 +394,13 @@ export const useSourcererDataView = (
  );
 };
 
+const detectionsPaths = [ALERTS_PATH, `${RULES_PATH}/id/:id`, `${CASES_PATH}/:detailName`];
+
 export const getScopeFromPath = (
  pathname: string
 ): SourcererScopeName.default | SourcererScopeName.detections =>
  matchPath(pathname, {
- path: [ALERTS_PATH, `${RULES_PATH}/id/:id`, `${CASES_PATH}/:detailName`],
+ path: detectionsPaths,
  strict: false,
  }) == null
  ? SourcererScopeName.default

@@ -0,0 +1,51 @@
+# Sourcerer Container
+
+### `useInitSourcerer`
+ - called at the top of the app in HomePageComponent to initialize the sourcerer state!
+ - Calls `useSourcererDataView` (see below) for the active scope (ex: `SourcererScopeName.default | SourcererScopeName.detections`)
+ - If there is an error with the data view, thrown here
+ - Run index field search for the active data view id, and when the active data view id updates 
+ - we changed the logic to not fetch all the index fields for every data view on the loading of the app because user can have a lot of them and it can slow down the loading of the app and maybe blow up the memory of the browser. 
+ - We decided to load the data view patternList and fields on demand, we know that will only have to load this data view on default and timeline scope. We will use two conditions to see if we need to fetch and initialize the data view selected. First, we will make sure that we did not already fetch them by using `searchedIds` and then we will init them if `selectedPatterns` and `missingPatterns` are empty.
+ - onSignalIndexUpdated
+ - called when signal index first has data in order to add it to the defaultDataView and refresh the index fields
+
+### `useSourcererDataView` 
+ - returns combined data from SourcererDataView and SourcererScope to create SelectedDataView state
+```typescript
+interface SelectedDataView {
+ browserFields: SourcererDataView['browserFields'];
+ dataViewId: string | null; // null if legacy pre-8.0 timeline
+ docValueFields: SourcererDataView['docValueFields'];
+ /**
+ * DataViewBase with enhanced index fields used in timelines
+ */
+ indexPattern: SecuritySolutionDataViewBase;
+ /** do the selected indices exist */
+ indicesExist: boolean;
+ /** is an update being made to the data view */
+ loading: boolean;
+ /** all active & inactive patterns from SourcererDataView['title'] */
+ patternList: string[];
+ runtimeMappings: SourcererDataView['runtimeMappings'];
+ /** all selected patterns from SourcererScope['selectedPatterns'] */
+ selectedPatterns: SourcererScope['selectedPatterns'];
+ // active patterns when dataViewId == null
+ activePatterns?: string[];
+}
+```
+
+### `useDataView`
+- called to get `indexFieldSearch`, which gets the fields and formats them in `getDataViewStateFromIndexFields` in `use_data_view.tsx`
+- `indexFieldSearch` calls the `IndexFieldsStrategyRequest` in the `timelines` plugin. This request takes an argument of either `dataViewId` or `indices` to get the fields. 
+ - Our app uses `dataViewId`, getting the fields from a Data View that includes runtime fields. No matter what the sourcerer indices are set to, we will get the same fields always from the Data View. If we only requested `indices`, we would get just the fields for those indices (not all like Data View) and therefore would not get the runtime fields. So even when we are in `SourcererScopeName.detections` narrowed down to just `.alerts-security.alerts-default`, we get all the fields from the entire Security Solution Data View.
+
+### `useSignalHelpers`
+ - called on from detections and timelines scopes
+ - `signalIndexNeedsInit` - when defined, signal index has been initiated but does not exist
+ - `pollForSignalIndex` - when false, signal index has been initiated
+
+
+### Adding sourcerer to a new page
+- In order for the sourcerer to show up on a page, it needs to be added to the array `sourcererPaths`
+- The scope of a sourcerer component will be default unless the path is added to the `detectionsPaths` array, in which case the scope can be detections
@@ -60,9 +60,12 @@ export interface KibanaDataView {
  */
 export interface SourcererDataView extends KibanaDataView {
  id: string;
- /** we need this for @timestamp data */
+ /** determines how we can use the field in the app
+ * aggregatable, searchable, type, example
+ * category, description, format
+ * indices the field is included in etc*/
  browserFields: BrowserFields;
- /** we need this for @timestamp data */
+ /** query DSL field and format */
  docValueFields: DocValueFields[];
  /** comes from dataView.fields.toSpec() */
  indexFields: SecuritySolutionDataViewBase['fields'];

@@ -0,0 +1,83 @@
+# Sourcerer Redux
+
+Sourcerer model for redux
+```typescript
+interface SourcererModel {
+ /** default security-solution data view */
+ defaultDataView: SourcererDataView & { id: string; error?: unknown };
+ /** all Kibana data views, including security-solution */
+ kibanaDataViews: SourcererDataView[];
+ /** security solution signals index name */
+ signalIndexName: string | null;
+ /** sourcerer scope data by id */
+ sourcererScopes: SourcererScopeById;
+}
+```
+
+The SourcererScopeName uniquely identifies a Sourcerer Scope. There are 3 in our app:
+```typescript
+enum SourcererScopeName {
+ default = 'default',
+ detections = 'detections',
+ timeline = 'timeline',
+}
+```
+
+Data related to each sourcerer scope
+```typescript
+interface SourcererScope {
+ /** Uniquely identifies a Sourcerer Scope */
+ id: SourcererScopeName;
+ /** is an update being made to the sourcerer data view */
+ loading: boolean;
+ /** selected data view id, null if it is legacy index patterns*/
+ selectedDataViewId: string | null;
+ /** selected patterns within the data view */
+ selectedPatterns: string[];
+ /** if has length,
+ * id === SourcererScopeName.timeline
+ * selectedDataViewId === null OR defaultDataView.id
+ * saved timeline has pattern that is not in the default */
+ missingPatterns: string[];
+}
+
+type SourcererScopeById = Record<SourcererScopeName, SourcererScope>;
+```
+
+```typescript
+interface KibanaDataView {
+ /** Uniquely identifies a Kibana Data View */
+ id: string;
+ /** list of active patterns that return data */
+ patternList: string[];
+ /**
+ * title of Kibana Data View
+ * title also serves as "all pattern list", including inactive
+ * comma separated string
+ */
+ title: string;
+}
+```
+
+KibanaDataView + timelines/index_fields enhanced field data
+```typescript
+interface SourcererDataView extends KibanaDataView {
+ id: string;
+ /** determines how we can use the field in the app
+ * aggregatable, searchable, type, example
+ * category, description, format
+ * indices the field is included in etc*/
+ browserFields: BrowserFields;
+ /** query DSL field and format */
+ docValueFields: DocValueFields[];
+ /** comes from dataView.fields.toSpec() */
+ indexFields: SecuritySolutionDataViewBase['fields'];
+ /** set when data view fields are fetched */
+ loading: boolean;
+ /**
+ * Needed to pass to search strategy
+ * Remove once issue resolved: https:/elastic/kibana/issues/111762
+ */
+ runtimeMappings: MappingRuntimeFields;
+}
+```
@@ -0,0 +1,78 @@
+# Sourcerer API
+
+### Model reference
+
+```typescript
+interface KibanaDataView {
+ /** Uniquely identifies a Kibana Data View */
+ id: string;
+ /** list of active patterns that return data */
+ patternList: string[];
+ /**
+ * title of Kibana Data View
+ * title also serves as "all pattern list", including inactive
+ * comma separated string
+ */
+ title: string;
+}
+```
+
+### API usage
+
+The sourcerer API has one route with 2 methods
+
+1. POST - `createSourcererDataViewRoute`
+ 1. REQUEST:
+ ```typescript
+ POST /internal/security_solution/sourcerer
+ {
+ patternList: [...configPatternList, ...(signal.name != null ? [signal.name] : [])]
+ }
+ ```
+ 2. RESPONSE:
+ ```typescript
+ {
+ /** default security-solution data view */
+ defaultDataView: KibanaDataView;
+
+ /** all Kibana data views, including default security-solution */
+ kibanaDataViews: KibanaDataView[];
+ }
+ ```
+ 3. This route is called from `security_solution/public/plugin.tsx` on app load. It passes an argument of `patternList` which is an array of the config index patterns defined in Stack Management > Advanced Settings > Security Solution > Elasticsearch indices along with the default signal index
+ 4. `dataViewService.getIdsWithTitle` is called to get all existing data views ids and titles
+ 5. Next `dataViewService.get` method is called to attempt to retrieve the default security data view by id (`siemClient.getSourcererDataViewId()`). If the data view id does not exist, it uses `dataViewService.createAndSave` to create the default security data view. 
+ 6. `patternListAsTitle` (a string of the patternList passed) is compared to the current `siemDataViewTitle`. If they do not match, we use `dataViewService.updateSavedObject` to update the data view title. This may happen when a pattern is added or removed from the Stack Management > Advanced Settings > Security Solution > Elasticsearch indices.
+ 7. Next we call `buildSourcererDataView` for the default data view only. This takes the `dataView.title` and finds which patterns on the list returns data. Valid patterns are returned in an array called `patternList`. The non-default data views will have an empty array for patternList, and we will call this function if/when the data view is selected to save time.
+ 8. At the end we return a body of 
+ ```
+ {
+ /** default security-solution data view */
+ defaultDataView: KibanaDataView;
+
+ /** all Kibana data views, including default security-solution */
+ kibanaDataViews: KibanaDataView[];
+ }
+ ```
+ 9. The other place this POST is called is when the default signal index does not yet exist. In the front-end there is a method called `pollForSignalIndex` that is defined when the signal index has been initiated but does not have data. It is called whenever the detection sourcerer or timeline sourcerer mounts, or whenever the search bar is refreshed. If the signal index is defined, `pollForSignalIndex` ceases to exist and is not called.
+ 10. One more place we call the POST method is when the signal index first has data, we send a POST in a method called `onSignalIndexUpdated` to include the newly created index in the data view
+2. GET - `getSourcererDataViewRoute`
+ 1. REQUEST:
+ ```typescript
+ GET /internal/security_solution/sourcerer?dataViewId=security-solution-default
+ ```
+ 2. RESPONSE:
+ ```typescript
+ KibanaDataView
+ ```
+3. When the user changes the data view from the default in the UI, we call the GET method to find which index patterns in the `dataView.title` are valid, returning a valid `patternList`. 
+4. We return a body of a single `KibanaDataView`
+
+### Helpers
+To build the valid pattern list, we call `findExistingIndices` which takes the pattern list as an argument, and returns a boolean array of which patterns are valid. To check if indices exist, we use the field caps API for each pattern checking for the field `_id`. This will return a list of valid indices. If the array is empty, no indices exist for the pattern. For example:
+```typescript
+ // Given
+ findExistingIndices(['auditbeat-*', 'fakebeat-*', 'packetbeat-*'])
+ // Returns
+ [true, false, true]
+```