rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB

docs/management/advanced-options.asciidoc

@@ -83,6 +83,10 @@ specific dashboard, application, or saved object as they enter each space.
 [[fields-popularlimit]]`fields:popularLimit`::
 The top N most popular fields to show.
 
+[[fileupload-maxfilesize]]`fileUpload:maxFileSize`::
+Sets the file size limit when importing files. The default
+value is `100MB`. The highest supported value for this setting is `1GB`.
+
 [[filtereditor-suggestvalues]]`filterEditor:suggestValues`::
 Set this property to `false` to prevent the filter editor from suggesting values
 for fields.
@@ -257,7 +261,7 @@ Hides the "Time" column in *Discover* and in all saved searches on dashboards.
 Highlights results in *Discover* and saved searches on dashboards. Highlighting
 slows requests when working on big documents.
 
-[[doctable-legacy]]`doc_table:legacy`:: 
+[[doctable-legacy]]`doc_table:legacy`::
 Controls the way the document table looks and works. Set this property to `true` to revert to the legacy implementation.
 
 [[discover-searchFieldsFromSource]]`discover:searchFieldsFromSource`::
@@ -281,10 +285,6 @@ must contain `from` and `to` values (see
 {ref}/common-options.html#date-math[accepted formats]). It is ignored unless
 `ml:anomalyDetection:results:enableTimeDefaults` is enabled.
 
-[[ml-filedatavisualizermaxfilesize]]`ml:fileDataVisualizerMaxFileSize`::
-Sets the file size limit when importing data in the {data-viz}. The default
-value is `100MB`. The highest supported value for this setting is `1GB`.
-
 
 [float]
 [[kibana-notification-settings]]

docs/user/ml/index.asciidoc

@@ -20,10 +20,10 @@ image::user/ml/images/ml-data-visualizer-sample.jpg[{data-viz} for sample flight
 experimental[] You can also upload a CSV, NDJSON, or log file. The *{data-viz}*
 identifies the file format and field mappings. You can then optionally import
 that data into an {es} index. To change the default file size limit, see
-<<kibana-ml-settings,Machine learning advanced settings>>.
+<<kibana-general-settings, fileUpload:maxFileSize advanced settings>>.
 
 If {stack-security-features} are enabled, users must have the necessary
-privileges to use {ml-features}. Refer to 
+privileges to use {ml-features}. Refer to
 {ml-docs}/setup.html#setup-privileges[Set up {ml-features}].
 
 NOTE: There are limitations in {ml-features} that affect {kib}. For more information, refer to {ml-docs}/ml-limitations.html[Machine learning].
@@ -40,15 +40,15 @@ false positives. {anomaly-detect-cap} runs in and scales with {es}, and
 includes an intuitive UI on the {kib} *Machine Learning* page for creating
 {anomaly-jobs} and understanding results.
 
-If you have a license that includes the {ml-features}, you can 
+If you have a license that includes the {ml-features}, you can
 create {anomaly-jobs} and manage jobs and {dfeeds} from the *Job Management*
-pane: 
+pane:
 
 [role="screenshot"]
 image::user/ml/images/ml-job-management.png[Job Management]
 
-You can use the *Settings* pane to create and edit 
-{ml-docs}/ml-calendars.html[calendars] and the filters that are used in 
+You can use the *Settings* pane to create and edit
+{ml-docs}/ml-calendars.html[calendars] and the filters that are used in
 {ml-docs}/ml-rules.html[custom rules]:
 
 [role="screenshot"]
@@ -69,13 +69,13 @@ occurring in your operational environment at that time:
 image::user/ml/images/ml-annotations-list.png[Single Metric Viewer with annotations]
 
 In some circumstances, annotations are also added automatically. For example, if
-the {anomaly-job} detects that there is missing data, it annotates the affected 
-time period. For more information, see 
-{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The 
+the {anomaly-job} detects that there is missing data, it annotates the affected
+time period. For more information, see
+{ml-docs}/ml-delayed-data-detection.html[Handling delayed data]. The
 *Job Management* pane shows the full list of annotations for each job.
 
-NOTE: The {kib} {ml-features} use pop-ups. You must configure your web 
-browser so that it does not block pop-up windows or create an exception for your 
+NOTE: The {kib} {ml-features} use pop-ups. You must configure your web
+browser so that it does not block pop-up windows or create an exception for your
 {kib} URL.
 
 For more information about the {anomaly-detect} feature, see
@@ -89,7 +89,7 @@ experimental[]
 
 The Elastic {ml} {dfanalytics} feature enables you to analyze your data using
 {classification}, {oldetection}, and {regression} algorithms and generate new
-indices that contain the results alongside your source data. 
+indices that contain the results alongside your source data.
 
 If you have a license that includes the {ml-features}, you can create
 {dfanalytics-jobs} and view their results on the *Data Frame Analytics* page in
@@ -98,5 +98,5 @@ If you have a license that includes the {ml-features}, you can create
 [role="screenshot"]
 image::user/ml/images/outliers.png[{oldetection-cap} results in {kib}]
 
-For more information about the {dfanalytics} feature, see 
-{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].
+For more information about the {dfanalytics} feature, see
+{ml-docs}/ml-dfanalytics.html[{ml-cap} {dfanalytics}].

src/core/server/ui_settings/saved_objects/migrations.test.ts

@@ -44,3 +44,37 @@ describe('ui_settings 7.9.0 migrations', () => {
  });
  });
 });
+
+describe('ui_settings 7.13.0 migrations', () => {
+ const migration = migrations['7.13.0'];
+
+ test('returns doc on empty object', () => {
+ expect(migration({} as SavedObjectUnsanitizedDoc)).toEqual({
+ references: [],
+ });
+ });
+ test('properly renames ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize', () => {
+ const doc = {
+ type: 'config',
+ id: '8.0.0',
+ attributes: {
+ buildNum: 9007199254740991,
+ 'ml:fileDataVisualizerMaxFileSize': '250MB',
+ },
+ references: [],
+ updated_at: '2020-06-09T20:18:20.349Z',
+ migrationVersion: {},
+ };
+ expect(migration(doc)).toEqual({
+ type: 'config',
+ id: '8.0.0',
+ attributes: {
+ buildNum: 9007199254740991,
+ 'fileUpload:maxFileSize': '250MB',
+ },
+ references: [],
+ updated_at: '2020-06-09T20:18:20.349Z',
+ migrationVersion: {},
+ });
+ });
+});

src/core/server/ui_settings/saved_objects/migrations.ts

@@ -28,4 +28,23 @@ export const migrations = {
  }),
  references: doc.references || [],
  }),
+ '7.13.0': (doc: SavedObjectUnsanitizedDoc<any>): SavedObjectSanitizedDoc<any> => ({
+ ...doc,
+ ...(doc.attributes && {
+ attributes: Object.keys(doc.attributes).reduce(
+ (acc, key) =>
+ key === 'ml:fileDataVisualizerMaxFileSize'
+ ? {
+ ...acc,
+ ['fileUpload:maxFileSize']: doc.attributes[key],
+ }
+ : {
+ ...acc,
+ [key]: doc.attributes[key],
+ },
+ {}
+ ),
+ }),
+ references: doc.references || [],
+ }),
 };

src/plugins/kibana_usage_collection/server/collectors/management/schema.ts

@@ -73,7 +73,7 @@ export const stackManagementSchema: MakeSchemaFrom<UsageStats> = {
  'discover:sort:defaultOrder': { type: 'keyword' },
  'context:step': { type: 'long' },
  'accessibility:disableAnimations': { type: 'boolean' },
- 'ml:fileDataVisualizerMaxFileSize': { type: 'keyword' },
+ 'fileUpload:maxFileSize': { type: 'keyword' },
  'ml:anomalyDetection:results:enableTimeDefaults': { type: 'boolean' },
  'ml:anomalyDetection:results:timeDefaults': { type: 'keyword' },
  'truncate:maxHeight': { type: 'long' },

src/plugins/kibana_usage_collection/server/collectors/management/types.ts

@@ -73,7 +73,7 @@ export interface UsageStats {
  'discover:sort:defaultOrder': string;
  'context:step': number;
  'accessibility:disableAnimations': boolean;
- 'ml:fileDataVisualizerMaxFileSize': string;
+ 'fileUpload:maxFileSize': string;
  'ml:anomalyDetection:results:enableTimeDefaults': boolean;
  'ml:anomalyDetection:results:timeDefaults': string;
  'truncate:maxHeight': number;

src/plugins/telemetry/schema/oss_plugins.json

@@ -4235,7 +4235,7 @@
  "accessibility:disableAnimations": {
  "type": "boolean"
  },
- "ml:fileDataVisualizerMaxFileSize": {
+ "fileUpload:maxFileSize": {
  "type": "keyword"
  },
  "ml:anomalyDetection:results:enableTimeDefaults": {

x-pack/plugins/file_upload/common/constants.ts

@@ -5,6 +5,8 @@
  * 2.0.
  */
 
+export const UI_SETTING_MAX_FILE_SIZE = 'fileUpload:maxFileSize';
+
 export const MB = Math.pow(2, 20);
 export const MAX_FILE_SIZE = '100MB';
 export const MAX_FILE_SIZE_BYTES = 104857600; // 100MB

x-pack/plugins/file_upload/kibana.json

@@ -4,5 +4,6 @@
  "kibanaVersion": "kibana",
  "server": true,
  "ui": true,
- "requiredPlugins": ["data", "usageCollection"]
+ "requiredPlugins": ["data", "usageCollection"],
+ "requiredBundles": ["kibanaReact"]
 }

x-pack/plugins/file_upload/public/api/index.ts

@@ -12,6 +12,8 @@ import type { IImporter, ImportFactoryOptions } from '../importer';
 export interface FileUploadStartApi {
  getFileUploadComponent(): Promise<React.ComponentType<FileUploadComponentProps>>;
  importerFactory(format: string, options: ImportFactoryOptions): Promise<IImporter | undefined>;
+ getMaxBytes(): number;
+ getMaxBytesFormatted(): string;
 }
 
 export async function getFileUploadComponent(): Promise<

x-pack/plugins/file_upload/public/components/geojson_file_picker.tsx

@@ -0,0 +1,162 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React, { Component } from 'react';
+import { Feature } from 'geojson';
+import { EuiFilePicker, EuiFormRow } from '@elastic/eui';
+import { i18n } from '@kbn/i18n';
+import { MB } from '../../common';
+import { getMaxBytesFormatted } from '../get_max_bytes';
+import { validateFile } from '../importer';
+import { GeoJsonImporter, GEOJSON_FILE_TYPES } from '../importer/geojson_importer';
+
+interface Props {
+ onSelect: ({
+ features,
+ geoFieldTypes,
+ importer,
+ indexName,
+ previewCoverage,
+ }: {
+ features: Feature[];
+ indexName: string;
+ importer: GeoJsonImporter;
+ geoFieldTypes: string[];
+ previewCoverage: number;
+ }) => void;
+ onClear: () => void;
+}
+
+interface State {
+ error: string | null;
+ isLoadingPreview: boolean;
+ previewSummary: string | null;
+}
+
+export class GeoJsonFilePicker extends Component<Props, State> {
+ private _isMounted = false;
+
+ state: State = {
+ error: null,
+ isLoadingPreview: false,
+ previewSummary: null,
+ };
+
+ async componentDidMount() {
+ this._isMounted = true;
+ }
+
+ componentWillUnmount() {
+ this._isMounted = false;
+ }
+
+ _onFileSelect = (files: FileList | null) => {
+ this.props.onClear();
+
+ this.setState({
+ error: null,
+ isLoadingPreview: false,
+ previewSummary: null,
+ });
+
+ if (files && files.length) {
+ this._loadFilePreview(files[0]);
+ }
+ };
+
+ async _loadFilePreview(file: File) {
+ this.setState({ isLoadingPreview: true });
+
+ let importer: GeoJsonImporter | null = null;
+ let previewError: string | null = null;
+ let preview: {
+ features: Feature[];
+ geoFieldTypes: string[];
+ previewCoverage: number;
+ } | null = null;
+ try {
+ validateFile(file, GEOJSON_FILE_TYPES);
+ importer = new GeoJsonImporter(file);
+ preview = await importer.previewFile(10000, MB * 3);
+ if (preview.features.length === 0) {
+ previewError = i18n.translate('xpack.fileUpload.geojsonFilePicker.noFeaturesDetected', {
+ defaultMessage: 'No GeoJson features found in selected file.',
+ });
+ }
+ } catch (error) {
+ previewError = error.message;
+ }
+
+ if (!this._isMounted) {
+ return;
+ }
+
+ this.setState({
+ error: previewError,
+ isLoadingPreview: false,
+ previewSummary:
+ !previewError && preview
+ ? i18n.translate('xpack.fileUpload.geojsonFilePicker.previewSummary', {
+ defaultMessage: 'Previewing {numFeatures} features, {previewCoverage}% of file.',
+ values: {
+ numFeatures: preview.features.length,
+ previewCoverage: preview.previewCoverage,
+ },
+ })
+ : null,
+ });
+
+ if (importer && preview) {
+ this.props.onSelect({
+ ...preview,
+ importer,
+ indexName: file.name.split('.')[0],
+ });
+ }
+ }
+
+ _renderHelpText() {
+ return this.state.previewSummary !== null ? (
+ this.state.previewSummary
+ ) : (
+ <span>
+ {i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedFormats', {
+ defaultMessage: 'Formats accepted: {fileTypes}',
+ values: { fileTypes: GEOJSON_FILE_TYPES.join(', ') },
+ })}
+ <br />
+ {i18n.translate('xpack.fileUpload.geojsonFilePicker.maxSize', {
+ defaultMessage: 'Max size: {maxFileSize}',
+ values: { maxFileSize: getMaxBytesFormatted() },
+ })}
+ <br />
+ {i18n.translate('xpack.fileUpload.geojsonFilePicker.acceptedCoordinateSystem', {
+ defaultMessage: 'Coordinates must be in EPSG:4326 coordinate reference system.',
+ })}
+ </span>
+ );
+ }
+
+ render() {
+ return (
+ <EuiFormRow
+ isInvalid={!!this.state.error}
+ error={!!this.state.error ? [this.state.error] : []}
+ helpText={this._renderHelpText()}
+ >
+ <EuiFilePicker
+ initialPromptText={i18n.translate('xpack.fileUpload.geojsonFilePicker.filePicker', {
+ defaultMessage: 'Select or drag and drop a file',
+ })}
+ onChange={this._onFileSelect}
+ accept={GEOJSON_FILE_TYPES.join(',')}
+ isLoading={this.state.isLoadingPreview}
+ />
+ </EuiFormRow>
+ );
+ }
+}