Merge branch 'main' into osquery-fix-saved-queries

docs/user/alerting/alerting-setup.asciidoc

@@ -5,35 +5,47 @@
 <titleabbrev>Set up</titleabbrev>
 ++++
 
-Alerting is automatically enabled in {kib}, but might require some additional configuration.
+Alerting is automatically enabled in {kib}, but might require some additional 
+configuration.
 
 [float]
 [[alerting-prerequisites]]
 === Prerequisites
 If you are using an *on-premises* Elastic Stack deployment:
 
-* In the kibana.yml configuration file, add the <<general-alert-action-settings,`xpack.encryptedSavedObjects.encryptionKey`>> setting.
-* For emails to have a footer with a link back to {kib}, set the <<server-publicBaseUrl, `server.publicBaseUrl`>> configuration setting.
+* In the kibana.yml configuration file, add the 
+<<general-alert-action-settings,`xpack.encryptedSavedObjects.encryptionKey`>> 
+setting.
+* For emails to have a footer with a link back to {kib}, set the 
+<<server-publicBaseUrl, `server.publicBaseUrl`>> configuration setting.
 
-If you are using an *on-premises* Elastic Stack deployment with <<using-kibana-with-security, *security*>>:
+If you are using an *on-premises* Elastic Stack deployment with 
+<<using-kibana-with-security, *security*>>:
 
-* If you are unable to access {kib} Alerting, ensure that you have not {ref}/security-settings.html#api-key-service-settings[explicitly disabled API keys].
+* If you are unable to access {kib} Alerting, ensure that you have not 
+{ref}/security-settings.html#api-key-service-settings[explicitly disabled API keys].
 
-The alerting framework uses queries that require the `search.allow_expensive_queries` setting to be `true`. See the scripts {ref}/query-dsl-script-query.html#_allow_expensive_queries_4[documentation]. 
+The alerting framework uses queries that require the 
+`search.allow_expensive_queries` setting to be `true`. See the scripts 
+{ref}/query-dsl-script-query.html#_allow_expensive_queries_4[documentation]. 
 
 [float]
 [[alerting-setup-production]]
 === Production considerations and scaling guidance
 
-When relying on alerting and actions as mission critical services, make sure you follow the <<alerting-production-considerations,Alerting production considerations>>.
+When relying on alerting and actions as mission critical services, make sure you 
+follow the 
+<<alerting-production-considerations,Alerting production considerations>>.
 
-See <<alerting-scaling-guidance>> for more information on the scalability of Alerting.
+See <<alerting-scaling-guidance>> for more information on the scalability of 
+Alerting.
 
 [float]
 [[alerting-security]]
 === Security
 
-To access alerting in a space, a user must have access to one of the following features:
+To access alerting in a space, a user must have access to one of the following 
+features:
 
 * Alerting
 * <<xpack-apm,*APM*>>
@@ -43,31 +55,53 @@ To access alerting in a space, a user must have access to one of the following f
 * <<xpack-siem,*Security*>>
 * <<uptime-app,*Uptime*>>
 
-See <<kibana-feature-privileges, feature privileges>> for more information on configuring roles that provide access to these features.
-Also note that a user will need +read+ privileges for the *Actions and Connectors* feature to attach actions to a rule or to edit a rule that has an action attached to it.
+See <<kibana-feature-privileges, feature privileges>> for more information on 
+configuring roles that provide access to these features.
+Also note that a user will need +read+ privileges for the 
+*Actions and Connectors* feature to attach actions to a rule or to edit a rule 
+that has an action attached to it.
 
 [float]
 [[alerting-restricting-actions]]
 ==== Restrict actions
 
-For security reasons you may wish to limit the extent to which {kib} can connect to external services. <<action-settings>> allows you to disable certain <<action-types>> and allowlist the hostnames that {kib} can connect with.
+For security reasons you may wish to limit the extent to which {kib} can connect 
+to external services. <<action-settings>> allows you to disable certain 
+<<action-types>> and allowlist the hostnames that {kib} can connect with.
 
 [float]
 [[alerting-spaces]]
 === Space isolation
 
-Rules and connectors are isolated to the {kib} space in which they were created. A rule or connector created in one space will not be visible in another. 
+Rules and connectors are isolated to the {kib} space in which they were created. 
+A rule or connector created in one space will not be visible in another. 
 
 [float]
 [[alerting-authorization]]
 === Authorization
 
-Rules are authorized using an <<api-keys,API key>> associated with the last user to edit the rule. This API key captures a snapshot of the user's privileges at the time of edit and is subsequently used to run all background tasks associated with the rule, including condition checks like {es} queries and triggered actions. The following rule actions will re-generate the API key:
+Rules are authorized using an <<api-keys,API key>> associated with the last user 
+to edit the rule. This API key captures a snapshot of the user's privileges at 
+the time of the edit. They are subsequently used to run all background tasks 
+associated with the rule, including condition checks like {es} queries and 
+triggered actions. The following rule actions will re-generate the API key:
 
 * Creating a rule
 * Updating a rule
 
+When you disable a rule, it retains the associated API key which is re-used when 
+the rule is enabled. If the API key is missing when you enable the rule (for 
+example, in the case of imported rules), it generates a new key that has your 
+security privileges.
+
+You can update an API key manually in 
+**{stack-manage-app} > {rules-ui}** or in the rule details page by selecting 
+**Update API key** in the actions menu.
+
 [IMPORTANT]
 ==============================================
-If a rule requires certain privileges, such as index privileges, to run, and a user without those privileges updates the rule, the rule will no longer function. Conversely, if a user with greater or administrator privileges modifies the rule, it will begin running with increased privileges.
+If a rule requires certain privileges, such as index privileges, to run, and a 
+user without those privileges updates the rule, the rule will no longer 
+function. Conversely, if a user with greater or administrator privileges 
+modifies the rule, it will begin running with increased privileges.
 ==============================================

packages/elastic-apm-synthtrace/src/index.ts

@@ -9,6 +9,7 @@
 export { timerange } from './lib/timerange';
 export { apm } from './lib/apm';
 export { stackMonitoring } from './lib/stack_monitoring';
+export { observer } from './lib/agent_config';
 export { cleanWriteTargets } from './lib/utils/clean_write_targets';
 export { createLogger, LogLevel } from './lib/utils/create_logger';
 

packages/elastic-apm-synthtrace/src/lib/agent_config/agent_config.ts

@@ -0,0 +1,24 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { AgentConfigFields } from './agent_config_fields';
+import { Metricset } from '../apm/metricset';
+
+export class AgentConfig extends Metricset<AgentConfigFields> {
+ constructor() {
+ super({
+ 'metricset.name': 'agent_config',
+ agent_config_applied: 1,
+ });
+ }
+
+ etag(etag: string) {
+ this.fields['labels.etag'] = etag;
+ return this;
+ }
+}

packages/elastic-apm-synthtrace/src/lib/agent_config/agent_config_fields.ts

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { ApmFields } from '../apm/apm_fields';
+
+export type AgentConfigFields = Pick<
+ ApmFields,
+ | '@timestamp'
+ | 'processor.event'
+ | 'processor.name'
+ | 'metricset.name'
+ | 'observer'
+ | 'ecs.version'
+ | 'event.ingested'
+> &
+ Partial<{
+ 'labels.etag': string;
+ agent_config_applied: number;
+ 'event.agent_id_status': string;
+ }>;

packages/elastic-apm-synthtrace/src/lib/agent_config/index.ts

@@ -0,0 +1,9 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+export { observer } from './observer';

packages/elastic-apm-synthtrace/src/lib/agent_config/observer.ts

@@ -0,0 +1,21 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { AgentConfigFields } from './agent_config_fields';
+import { AgentConfig } from './agent_config';
+import { Entity } from '../entity';
+
+export class Observer extends Entity<AgentConfigFields> {
+ agentConfig() {
+ return new AgentConfig();
+ }
+}
+
+export function observer() {
+ return new Observer({});
+}

packages/elastic-apm-synthtrace/src/lib/apm/instance.ts

@@ -45,7 +45,7 @@ export class Instance extends Entity<ApmFields> {
  }
 
  appMetrics(metrics: ApmApplicationMetricFields) {
- return new Metricset({
+ return new Metricset<ApmFields>({
  ...this.fields,
  'metricset.name': 'app',
  ...metrics,

packages/elastic-apm-synthtrace/src/lib/apm/metricset.ts

@@ -7,10 +7,10 @@
  */
 
 import { Serializable } from '../serializable';
-import { ApmFields } from './apm_fields';
+import { Fields } from '../entity';
 
-export class Metricset extends Serializable<ApmFields> {
- constructor(fields: ApmFields) {
+export class Metricset<TFields extends Fields> extends Serializable<TFields> {
+ constructor(fields: TFields) {
  super({
  'processor.event': 'metric',
  'processor.name': 'metric',

packages/elastic-apm-synthtrace/src/lib/stream_processor.ts

@@ -211,7 +211,9 @@ export class StreamProcessor<TFields extends Fields = ApmFields> {
  const eventType = d.processor.event as keyof ApmElasticsearchOutputWriteTargets;
  let dataStream = writeTargets[eventType];
  if (eventType === 'metric') {
- if (!d.service?.name) {
+ if (d.metricset?.name === 'agent_config') {
+ dataStream = 'metrics-apm.internal-default';
+ } else if (!d.service?.name) {
  dataStream = 'metrics-apm.app-default';
  } else {
  if (!d.transaction && !d.span) {

packages/elastic-apm-synthtrace/src/scripts/examples/04_agent_config.ts

@@ -0,0 +1,36 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { observer, timerange } from '../..';
+import { Scenario } from '../scenario';
+import { getLogger } from '../utils/get_common_services';
+import { RunOptions } from '../utils/parse_run_cli_flags';
+import { AgentConfigFields } from '../../lib/agent_config/agent_config_fields';
+
+const scenario: Scenario<AgentConfigFields> = async (runOptions: RunOptions) => {
+ const logger = getLogger(runOptions);
+
+ return {
+ generate: ({ from, to }) => {
+ const agentConfig = observer().agentConfig();
+
+ const range = timerange(from, to);
+ return range
+ .interval('30s')
+ .rate(1)
+ .generator((timestamp) => {
+ const events = logger.perf('generating_agent_config_events', () => {
+ return agentConfig.etag('test-etag').timestamp(timestamp);
+ });
+ return events;
+ });
+ },
+ };
+};
+
+export default scenario;

src/plugins/unified_search/public/actions/apply_filter_action.ts

@@ -10,7 +10,9 @@ import { i18n } from '@kbn/i18n';
 import { ThemeServiceSetup } from '@kbn/core/public';
 import { toMountPoint } from '@kbn/kibana-react-plugin/public';
 import { Action, createAction, IncompatibleActionError } from '@kbn/ui-actions-plugin/public';
-import { Filter, FilterManager, TimefilterContract, esFilters } from '@kbn/data-plugin/public';
+// for cleanup esFilters need to fix the issue https:/elastic/kibana/issues/131292
+import { FilterManager, TimefilterContract, esFilters } from '@kbn/data-plugin/public';
+import type { Filter } from '@kbn/es-query';
 import { getOverlays, getIndexPatterns } from '../services';
 import { applyFiltersPopover } from '../apply_filters';
 

src/plugins/unified_search/public/apply_filters/apply_filter_popover_content.tsx

@@ -24,7 +24,7 @@ import {
  mapAndFlattenFilters,
  getFieldDisplayValueFromFilter,
 } from '@kbn/data-plugin/public';
-import { Filter } from '@kbn/data-plugin/common';
+import type { Filter } from '@kbn/es-query';
 import { DataView } from '@kbn/data-views-plugin/public';
 import { FilterLabel } from '../filter_bar';
 

src/plugins/unified_search/public/apply_filters/apply_filters_popover.tsx

@@ -7,7 +7,7 @@
  */
 
 import React from 'react';
-import { Filter } from '@kbn/data-plugin/common';
+import type { Filter } from '@kbn/es-query';
 import { DataView } from '@kbn/data-views-plugin/common';
 
 type CancelFnType = () => void;

src/plugins/unified_search/public/autocomplete/providers/kql_query_suggestion/conjunction.test.ts

@@ -7,7 +7,7 @@
  */
 
 import { coreMock } from '@kbn/core/public/mocks';
-import { KueryNode } from '@kbn/data-plugin/public';
+import type { KueryNode } from '@kbn/es-query';
 import { setupGetConjunctionSuggestions } from './conjunction';
 import { QuerySuggestionGetFnArgs } from '../query_suggestion_provider';
 

src/plugins/unified_search/public/autocomplete/providers/kql_query_suggestion/field.test.ts

@@ -8,7 +8,8 @@
 
 import indexPatternResponse from './__fixtures__/index_pattern_response.json';
 
-import { indexPatterns as indexPatternsUtils, KueryNode } from '@kbn/data-plugin/public';
+import { indexPatterns as indexPatternsUtils } from '@kbn/data-plugin/public';
+import type { KueryNode } from '@kbn/es-query';
 import { setupGetFieldSuggestions } from './field';
 import { QuerySuggestionGetFnArgs } from '../query_suggestion_provider';
 import { coreMock } from '@kbn/core/public/mocks';

src/plugins/unified_search/public/autocomplete/providers/kql_query_suggestion/field.tsx

@@ -6,6 +6,7 @@
  * Side Public License, v 1.
  */
 
+// for replace IFieldType => DataViewField need to fix the issue https:/elastic/kibana/issues/131292
 import { IFieldType, indexPatterns as indexPatternsUtils } from '@kbn/data-plugin/public';
 import { flatten } from 'lodash';
 import { sortPrefixFirst } from './sort_prefix_first';

src/plugins/unified_search/public/autocomplete/providers/kql_query_suggestion/operator.test.ts

@@ -9,7 +9,7 @@
 import indexPatternResponse from './__fixtures__/index_pattern_response.json';
 
 import { setupGetOperatorSuggestions } from './operator';
-import { KueryNode } from '@kbn/data-plugin/public';
+import type { KueryNode } from '@kbn/es-query';
 import { QuerySuggestionGetFnArgs } from '../query_suggestion_provider';
 import { coreMock } from '@kbn/core/public/mocks';
 

src/plugins/unified_search/public/autocomplete/providers/kql_query_suggestion/value.test.ts

@@ -10,7 +10,7 @@ import { setupGetValueSuggestions } from './value';
 import indexPatternResponse from './__fixtures__/index_pattern_response.json';
 
 import { coreMock } from '@kbn/core/public/mocks';
-import { KueryNode } from '@kbn/data-plugin/public';
+import type { KueryNode } from '@kbn/es-query';
 import { QuerySuggestionGetFnArgs } from '../query_suggestion_provider';
 
 const mockKueryNode = (kueryNode: Partial<KueryNode>) => kueryNode as unknown as KueryNode;

src/plugins/unified_search/public/autocomplete/providers/kql_query_suggestion/value.ts

@@ -8,7 +8,9 @@
 
 import { flatten } from 'lodash';
 import { CoreSetup } from '@kbn/core/public';
-import { IFieldType, IIndexPattern } from '@kbn/data-plugin/public';
+// for replace IIndexPattern => DataView and IFieldType => DataViewField
+// need to fix the issue https:/elastic/kibana/issues/131292
+import type { IIndexPattern, IFieldType } from '@kbn/data-views-plugin/common';
 import { escapeQuotes } from './lib/escape_kuery';
 import { KqlQuerySuggestionProvider } from './types';
 import type { UnifiedSearchPublicPluginStart } from '../../../types';

src/plugins/unified_search/public/autocomplete/providers/query_suggestion_provider.ts

@@ -7,7 +7,8 @@
  */
 
 import { ValueSuggestionsMethod } from '@kbn/data-plugin/common';
-import { IFieldType, IIndexPattern } from '@kbn/data-plugin/common';
+// for replace IIndexPattern => DataView need to fix the issue https:/elastic/kibana/issues/131292
+import type { DataViewField, IIndexPattern } from '@kbn/data-views-plugin/common';
 
 export enum QuerySuggestionTypes {
  Field = 'field',
@@ -47,7 +48,7 @@ export interface QuerySuggestionBasic {
 /** @public **/
 export interface QuerySuggestionField extends QuerySuggestionBasic {
  type: QuerySuggestionTypes.Field;
- field: IFieldType;
+ field: DataViewField;
 }
 
 /** @public **/

src/plugins/unified_search/public/autocomplete/providers/value_suggestion_provider.ts

@@ -9,12 +9,10 @@
 import { CoreSetup } from '@kbn/core/public';
 import dateMath from '@kbn/datemath';
 import { memoize } from 'lodash';
-import {
- IIndexPattern,
- IFieldType,
- UI_SETTINGS,
- ValueSuggestionsMethod,
-} from '@kbn/data-plugin/common';
+import { UI_SETTINGS, ValueSuggestionsMethod } from '@kbn/data-plugin/common';
+// for replace IIndexPattern => DataView and IFieldType => DataViewField
+// need to fix the issue https:/elastic/kibana/issues/131292
+import type { IIndexPattern, IFieldType } from '@kbn/data-views-plugin/common';
 import type { TimefilterSetup } from '@kbn/data-plugin/public';
 import { AutocompleteUsageCollector } from '../collectors';