HTTP CRUD+ API for Index Patterns

[streamich]

Summary

Closes #72360
Closes #12756

Adds REST API for index patterns—a wrapper around the Index Patterns service.

• Index Pattern CRUD API
  • Create — POST /api/index_patterns/index_pattern
  • Read — GET /api/index_patterns/index_pattern/{id}
  • Delete — DELETE /api/index_patterns/index_pattern/{id}
  • Update portion of an index pattern — POST /api/index_patterns/index_pattern/{id}
    • title
    • timeFieldName
    • intervalName
    • fields
      • Optionally refresh fields.
    • sourceFilters
    • fieldFormatMap
    • type
    • typeMeta
• Fields API — update or remove existing or non-existing field metadata (count, customLabel, format).
  • Put field metadata — PUT /api/index_patterns/index_pattern/{id}/fields
• Scripted Fields API
  • Create a new scripted field — POST /api/index_patterns/index_pattern/{id}/scripted_field
  • Read a scripted field — GET /api/index_patterns/index_pattern/{id}/scripted_field/{name}
  • Update a an existing scripted field — POST /api/index_patterns/index_pattern/{id}/scripted_field/{name}
  • Upsert a scripted field — PUT /api/index_patterns/index_pattern/{id}/scripted_field
  • Remove a scripted field — DELETE /api/index_patterns/index_pattern/{id}/scripted_field/{name}

Other:

• Integration tests

---

See Postman API collection

{
	"info": {
		"_postman_id": "c01b4f06-1a9a-4625-bb7e-8cc70000922c",
		"name": "Index Patterns",
		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
	},
	"item": [
		{
			"name": "Fields API",
			"item": [
				{
					"name": "Create field",
					"request": {
						"method": "POST",
						"header": [
							{
								"key": "Content-Type",
								"name": "Content-Type",
								"type": "text",
								"value": "application/json"
							},
							{
								"key": "kbn-xsrf",
								"type": "text",
								"value": "true"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n    \"field\": {\n        \"name\": \"test3\"\n    }\n}\n"
						},
						"url": {
							"raw": "{{origin}}/api/index_patterns/index_pattern/ab2fb6d0-2b49-11eb-8840-4d2b9eed180a/field",
							"host": [
								"{{origin}}"
							],
							"path": [
								"api",
								"index_patterns",
								"index_pattern",
								"ab2fb6d0-2b49-11eb-8840-4d2b9eed180a",
								"field"
							]
						}
					},
					"response": []
				},
				{
					"name": "Upsert field",
					"request": {
						"method": "PUT",
						"header": [
							{
								"key": "Content-Type",
								"name": "Content-Type",
								"type": "text",
								"value": "application/json"
							},
							{
								"key": "kbn-xsrf",
								"type": "text",
								"value": "true"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n    \"field\": {\n        \"name\": \"test4\",\n        \"type\": \"number\"\n    }\n}\n"
						},
						"url": {
							"raw": "{{origin}}/api/index_patterns/index_pattern/ab2fb6d0-2b49-11eb-8840-4d2b9eed180a/field",
							"host": [
								"{{origin}}"
							],
							"path": [
								"api",
								"index_patterns",
								"index_pattern",
								"ab2fb6d0-2b49-11eb-8840-4d2b9eed180a",
								"field"
							]
						}
					},
					"response": []
				},
				{
					"name": "Read field",
					"protocolProfileBehavior": {
						"disableBodyPruning": true
					},
					"request": {
						"method": "GET",
						"header": [
							{
								"key": "Content-Type",
								"name": "Content-Type",
								"type": "text",
								"value": "application/json"
							},
							{
								"key": "kbn-xsrf",
								"type": "text",
								"value": "true"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n    \"field\": {\n        \"name\": \"test4\",\n        \"type\": \"number\"\n    }\n}\n"
						},
						"url": {
							"raw": "{{origin}}/api/index_patterns/index_pattern/ab2fb6d0-2b49-11eb-8840-4d2b9eed180a/field/test4",
							"host": [
								"{{origin}}"
							],
							"path": [
								"api",
								"index_patterns",
								"index_pattern",
								"ab2fb6d0-2b49-11eb-8840-4d2b9eed180a",
								"field",
								"test4"
							]
						}
					},
					"response": []
				},
				{
					"name": "Update field",
					"request": {
						"method": "POST",
						"header": [
							{
								"key": "Content-Type",
								"name": "Content-Type",
								"type": "text",
								"value": "application/json"
							},
							{
								"key": "kbn-xsrf",
								"type": "text",
								"value": "true"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n    \"field\": {\n        \"count\": 0\n    }\n}"
						},
						"url": {
							"raw": "{{origin}}/api/index_patterns/index_pattern/ab2fb6d0-2b49-11eb-8840-4d2b9eed180a/field/test4",
							"host": [
								"{{origin}}"
							],
							"path": [
								"api",
								"index_patterns",
								"index_pattern",
								"ab2fb6d0-2b49-11eb-8840-4d2b9eed180a",
								"field",
								"test4"
							]
						}
					},
					"response": []
				},
				{
					"name": "Delete field",
					"request": {
						"method": "DELETE",
						"header": [
							{
								"key": "Content-Type",
								"name": "Content-Type",
								"type": "text",
								"value": "application/json"
							},
							{
								"key": "kbn-xsrf",
								"type": "text",
								"value": "true"
							}
						],
						"body": {
							"mode": "raw",
							"raw": "{\n    \"field\": {\n        \"name\": \"test4\",\n        \"type\": \"number\"\n    }\n}\n"
						},
						"url": {
							"raw": "{{origin}}/api/index_patterns/index_pattern/ab2fb6d0-2b49-11eb-8840-4d2b9eed180a/field/test4",
							"host": [
								"{{origin}}"
							],
							"path": [
								"api",
								"index_patterns",
								"index_pattern",
								"ab2fb6d0-2b49-11eb-8840-4d2b9eed180a",
								"field",
								"test4"
							]
						}
					},
					"response": []
				}
			]
		},
		{
			"name": "Create index pattern (simple)",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"value": "application/json",
						"type": "text"
					},
					{
						"key": "kbn-xsrf",
						"value": "true",
						"type": "text"
					}
				],
				"body": {
					"mode": "raw",
					"raw": "{\n    \"index_pattern\": {\n        \"title\": \"test\"\n    }\n}\n"
				},
				"url": {
					"raw": "{{origin}}/api/index_patterns/index_pattern",
					"host": [
						"{{origin}}"
					],
					"path": [
						"api",
						"index_patterns",
						"index_pattern"
					]
				}
			},
			"response": []
		},
		{
			"name": "Update index pattern (title)",
			"request": {
				"method": "POST",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/json"
					},
					{
						"key": "kbn-xsrf",
						"value": "true",
						"type": "text"
					}
				],
				"body": {
					"mode": "raw",
					"raw": "{\n    \"index_pattern\": {\n        \"title\": null\n    }\n}\n"
				},
				"url": {
					"raw": "{{origin}}/api/index_patterns/index_pattern/e1a2d0d0-2b3f-11eb-87a6-8b5177386415",
					"host": [
						"{{origin}}"
					],
					"path": [
						"api",
						"index_patterns",
						"index_pattern",
						"e1a2d0d0-2b3f-11eb-87a6-8b5177386415"
					]
				}
			},
			"response": []
		},
		{
			"name": "Read index pattern",
			"protocolProfileBehavior": {
				"disableBodyPruning": true
			},
			"request": {
				"method": "GET",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/json"
					}
				],
				"body": {
					"mode": "raw",
					"raw": "{\n    \"index_pattern\": {\n        \"title\": \"foo\"\n    }\n}\n"
				},
				"url": {
					"raw": "{{origin}}/api/index_patterns/index_pattern/e1a2d0d0-2b3f-11eb-87a6-8b5177386415",
					"host": [
						"{{origin}}"
					],
					"path": [
						"api",
						"index_patterns",
						"index_pattern",
						"e1a2d0d0-2b3f-11eb-87a6-8b5177386415"
					]
				}
			},
			"response": []
		},
		{
			"name": "Delete index pattern",
			"request": {
				"method": "DELETE",
				"header": [
					{
						"key": "Content-Type",
						"name": "Content-Type",
						"type": "text",
						"value": "application/json"
					},
					{
						"key": "kbn-xsrf",
						"value": "true",
						"type": "text"
					}
				],
				"body": {
					"mode": "raw",
					"raw": "{\n    \"index_pattern\": {\n        \"title\": \"foo\"\n    }\n}\n"
				},
				"url": {
					"raw": "{{origin}}/api/index_patterns/index_pattern/c03a9d60-2b35-11eb-bbe5-9f48d9a2a98f",
					"host": [
						"{{origin}}"
					],
					"path": [
						"api",
						"index_patterns",
						"index_pattern",
						"c03a9d60-2b35-11eb-bbe5-9f48d9a2a98f"
					]
				}
			},
			"response": []
		}
	],
	"auth": {
		"type": "basic",
		"basic": [
			{
				"key": "password",
				"value": "changeme",
				"type": "string"
			},
			{
				"key": "username",
				"value": "elastic",
				"type": "string"
			}
		]
	},
	"event": [
		{
			"listen": "prerequest",
			"script": {
				"id": "ea2d6011-8cc5-4123-94e8-317e44ca1e00",
				"type": "text/javascript",
				"exec": [
					""
				]
			}
		},
		{
			"listen": "test",
			"script": {
				"id": "3d326b30-1ad1-4079-a160-388876d120ef",
				"type": "text/javascript",
				"exec": [
					""
				]
			}
		}
	],
	"variable": [
		{
			"id": "0602ae6f-3ba7-4639-9bfc-e091521d6204",
			"key": "origin",
			"value": "",
			"type": "string"
		}
	]
}

---

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

---

Release note

Index Patterns service now has public HTTP API, which third parties can use to manage index patterns, index pattern field metadata, and scripted fields programatically. API consists of:

• Index Patterns API
  • Create an index pattern — POST /api/index_patterns/index_pattern
  • Fetch an index pattern by {id} — GET /api/index_patterns/index_pattern/{id}
  • Delete an index pattern by {id} — DELETE /api/index_patterns/index_pattern/{id}
  • Partially update an index pattern by {id} — POST /api/index_patterns/index_pattern/{id}
• Fields API
  • Update field — POST /api/index_patterns/index_pattern/{id}/fields
• Scripted Fields API
  • Create a scripted field — POST /api/index_patterns/index_pattern/{id}/scripted_field
  • Upsert a scripted field — PUT /api/index_patterns/index_pattern/{id}/scripted_field
  • Fetch a scripted field — GET /api/index_patterns/index_pattern/{id}/scripted_field/{name}
  • Remove a scripted field — DELETE /api/index_patterns/index_pattern/{id}/scripted_field/{name}
  • Update a scripted field — POST /api/index_patterns/index_pattern/{id}/scripted_field/{name}

---

Documentation

Index Patterns HTTP API

Index patterns provide Rest-like HTTP CRUD+ API with the following endpoints:

• Index Patterns API
  • Create an index pattern — POST /api/index_patterns/index_pattern
  • Fetch an index pattern by {id} — GET /api/index_patterns/index_pattern/{id}
  • Delete an index pattern by {id} — DELETE /api/index_patterns/index_pattern/{id}
  • Partially update an index pattern by {id} — POST /api/index_patterns/index_pattern/{id}
• Fields API
  • Update field — POST /api/index_patterns/index_pattern/{id}/fields
• Scripted Fields API
  • Create a scripted field — POST /api/index_patterns/index_pattern/{id}/scripted_field
  • Upsert a scripted field — PUT /api/index_patterns/index_pattern/{id}/scripted_field
  • Fetch a scripted field — GET /api/index_patterns/index_pattern/{id}/scripted_field/{name}
  • Remove a scripted field — DELETE /api/index_patterns/index_pattern/{id}/scripted_field/{name}
  • Update a scripted field — POST /api/index_patterns/index_pattern/{id}/scripted_field/{name}

Index Patterns API

Index Patterns CURD API allows you to create, retrieve and delete index patterns. I also
exposes an update endpoint which allows you to update specific fields without changing
the rest of the index pattern object.

Create an index pattern

Create an index pattern with a custom title.

POST /api/index_patterns/index_pattern
{
    "index_pattern": {
        "title": "hello"
    }
}

Customize creation behavior with:

• override — if set to true, replaces an existing index pattern if an
  index pattern with the provided title already exists. Defaults to false.
• refresh_fields — if set to true reloads index pattern fields after
  the index pattern is stored. Defaults to false.

POST /api/index_patterns/index_pattern
{
    "override": false,
    "refresh_fields": true,
    "index_pattern": {
        "title": "hello"
    }
}

At creation all index pattern fields are option and you can provide them.

POST /api/index_patterns/index_pattern
{
    "index_pattern": {
        "id": "...",
        "version": "...",
        "title": "...",
        "type": "...",
        "intervalName": "...",
        "timeFieldName": "...",
        "sourceFilters": [],
        "fields": {},
        "typeMeta": {},
        "fieldFormats": {},
        "fieldAttrs": {}
    }
}

The endpoint returns the created index pattern object.

{
    "index_pattern": {}
}

Fetch an index pattern by ID

Retrieve and index pattern by its ID.

GET /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Returns an index pattern object.

{
    "index_pattern": {
        "id": "...",
        "version": "...",
        "title": "...",
        "type": "...",
        "intervalName": "...",
        "timeFieldName": "...",
        "sourceFilters": [],
        "fields": {},
        "typeMeta": {},
        "fieldFormats": {},
        "fieldAttrs": {}
    }
}

Delete an index pattern by ID

Delete and index pattern by its ID.

DELETE /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Returns an '200 OK` response with empty body on success.

Partially update an index pattern by ID

Update part of an index pattern. Only provided fields will be updated on the
index pattern, missing fields will stay as they are persisted.

These fields can be update partially:

• title
• timeFieldName
• intervalName
• fields (optionally refresh fields)
• sourceFilters
• fieldFormatMap
• type
• typeMeta

Update a title of an index pattern.

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
{
    "index_pattern": {
        "title": "new_title"
    }
}

All update fields are optional, you can specify the following fields.

POST /api/index_patterns/index_pattern
{
    "index_pattern": {
        "title": "...",
        "timeFieldName": "...",
        "intervalName": "...",
        "sourceFilters": [],
        "fieldFormats": {},
        "type": "...",
        "typeMeta": {},
        "fields": {}
    }
}

• refresh_fields — if set to true reloads index pattern fields after
  the index pattern is stored. Defaults to false.

POST /api/index_patterns/index_pattern
{
    "refresh_fields": true,
    "index_pattern": {
        "fields": {}
    }
}

This endpoint returns the updated index pattern object.

{
    "index_pattern": {

    }
}

Fields API

Fields API allows to change field metadata, such as count, customLabel, and format.

Update fields

Update endpoint allows you to update fields presentation metadata, such as count,
customLabel, and format. You can update multiple fields in one request. Updates
are merges with persisted metadata. To remove existing metadata specify null as value.

Set popularity count for field foo:

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
{
    "fields": {
        "foo": {
            "count": 123
        }
    }
}

Update multiple metadata values and fields in one request:

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
{
    "fields": {
        "foo": {
            "count": 123,
            "customLabel": "Foo"
        },
        "bar": {
            "customLabel": "Bar"
        }
    }
}

Use null value to delete metadata:

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
{
    "fields": {
        "foo": {
            "customLabel": null
        }
    }
}

This endpoint returns the updated index pattern object.

{
    "index_pattern": {

    }
}

Scripted Fields API

Scripted Fields API provides CRUD API for scripted fields of an index pattern.

Create a scripted field

Create a field by simply specifying its name, will default to string type. Returns
an error if a field with the provided name already exists.

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
{
    "field": {
        "name": "my_field"
    }
}

Create a field by specifying all field properties.

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
{
    "field": {
        "name": "",
        "type": "",
        "searchable": false,
        "aggregatable": false,
        "count": 0,
        "script": "",
        "scripted": false,
        "lang": "",
        "conflictDescriptions": {},
        "format": {},
        "esTypes": [],
        "readFromDocValues": false,
        "subType": {},
        "indexed": false,
        "customLabel": "",
        "shortDotsEnable": false
    }
}

Upsert a scripted field

Creates a new field or updates an existing one, if one already exists with the same name.

Create a field by simply specifying its name.

PUT /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
{
    "field": {
        "name": "my_field"
    }
}

Create a field by specifying all field properties.

PUT /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
{
    "field": {
        "name": "",
        "type": "",
        "searchable": false,
        "aggregatable": false,
        "count": 0,
        "script": "",
        "scripted": false,
        "lang": "",
        "conflictDescriptions": {},
        "format": {},
        "esTypes": [],
        "readFromDocValues": false,
        "subType": {},
        "indexed": false,
        "customLabel": "",
        "shortDotsEnable": false
    }
}

Fetch a scripted field

Fetch an existing index pattern field by field name.

GET /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>

Returns the field object.

{
    "field": {}
}

Delete a scripted field

Delete a field of an index pattern.

DELETE /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>

Update a an existing scripted field

Updates an exiting field by mergin provided properties with the existing ones. If
there is no existing field with the specified name, returns a 404 Not Found error.

You can specify any field properties, except name which is specified in the URL path.

POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
{
    "field": {
        "type": "",
        "searchable": false,
        "aggregatable": false,
        "count": 0,
        "script": "",
        "scripted": false,
        "lang": "",
        "conflictDescriptions": {},
        "format": {},
        "esTypes": [],
        "readFromDocValues": false,
        "subType": {},
        "indexed": false,
        "customLabel": "",
        "shortDotsEnable": false
    }
}

[mattkime]

A few items to be addressed

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 28086cb

Metrics [docs]

Distributable file count

id	before	after	diff
default	46920	47693	+773
oss	27565	27578	+13

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	987.9KB	988.9KB	+1002.0B

History

• 💔 Build #92389 failed 87732fb
• 💚 Build #92337 succeeded 2bb780f
• 💔 Build #92002 failed b6d49f8
• 💚 Build #91684 succeeded cce1c9a
• 💔 Build #91639 failed 9dbe3f9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream