Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a template parameter to override auto_create_index value #61858

Merged
merged 78 commits into from
Oct 26, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
ee50924
Override auto-create index behaviour via a template flag
pugnascotia Aug 3, 2020
f62c83b
WIP - trying to test watcher
pugnascotia Aug 4, 2020
097d9f5
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 1, 2020
3bb580c
Fixes
pugnascotia Sep 2, 2020
42cfc00
Remove logging
pugnascotia Sep 2, 2020
8c17371
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 2, 2020
9a1e10a
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 4, 2020
961cfcf
Use AutoCreateIndex in AutoCreateAction
pugnascotia Sep 9, 2020
41fdcb3
Imports
pugnascotia Sep 10, 2020
2228033
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 10, 2020
b00297e
Fixes
pugnascotia Sep 14, 2020
ad5021d
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 14, 2020
ad9e699
Fix compile error
pugnascotia Sep 14, 2020
c42e3ce
Tweak ComposableIndexTemplate serialisation under code changes land i…
pugnascotia Sep 14, 2020
6a689b2
Fix auto create index bug
pugnascotia Sep 16, 2020
f81a510
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 16, 2020
2d54d2c
Test fix
pugnascotia Sep 16, 2020
85abcbc
Handle null Boolean
pugnascotia Sep 16, 2020
eb299a0
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 18, 2020
a3e3691
Imports
pugnascotia Sep 18, 2020
9421140
Checkstyle
pugnascotia Sep 18, 2020
ebb42d4
Tweaks
pugnascotia Sep 18, 2020
a93cfb2
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 18, 2020
bfe8c8c
Post-merge fix
pugnascotia Sep 18, 2020
2694fb2
Update synthentics template to add allow_auto_create
pugnascotia Sep 18, 2020
5d5eeb4
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 22, 2020
39abb38
Address review feedback
pugnascotia Sep 22, 2020
4b9034a
Address review feedback
pugnascotia Sep 22, 2020
ce8c2ed
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 22, 2020
2e9a4c7
Specify the same version for ser/deser
pugnascotia Sep 22, 2020
d5db431
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 24, 2020
1d07709
Fix stupid typo
pugnascotia Sep 24, 2020
36b9ca8
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Sep 28, 2020
93e2440
Address review comments
pugnascotia Sep 28, 2020
4a2e64c
Increment some template versions due to the new setting
pugnascotia Sep 28, 2020
82ad6e4
Add Java REST tests for auto-creating indices
pugnascotia Sep 28, 2020
6022b3c
Checkstyle
pugnascotia Sep 28, 2020
41b2fc6
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Oct 12, 2020
6127caf
Update component template docs
pugnascotia Oct 12, 2020
9d6b6cf
Checkstyle
pugnascotia Oct 12, 2020
c0394e4
Docs tweak
pugnascotia Oct 13, 2020
cbfbb98
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Oct 13, 2020
383d6ff
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Oct 19, 2020
61ab28e
Make data stream subject to auto_create_index
pugnascotia Oct 19, 2020
6c4c817
Add tests for auto-creating data streams
pugnascotia Oct 19, 2020
ed692c7
Formatting
pugnascotia Oct 19, 2020
e8617b6
Merge branch 'master' into 20640-auto-create-templates
elasticmachine Oct 19, 2020
4f051af
Fix max/min aggs for unsigned_long (#63904)
mayya-sharipova Oct 19, 2020
44d7d66
Add APM configuration index to Kibana system indices (#63756)
williamrandolph Oct 19, 2020
e145266
[ML] Rename evaluation metric result fields to value (#63809)
dimitris-athanasiou Oct 20, 2020
9e8221a
Update gradle wrapper to Gradle 6.7 (#62386)
breskeby Oct 20, 2020
dd7a22f
Add snapshot shard size based test in DiskThresholdDeciderTests (#63546)
tlrx Oct 20, 2020
cfca1e4
DocumentMapperParser to no longer depend directly on MapperService (#…
javanna Oct 20, 2020
85c0b54
Composite aggregation must check live docs when the index is sorted (…
jimczi Oct 20, 2020
0204f0a
Fix build tools integTest failure after vault update (#63914)
breskeby Oct 20, 2020
eae0eaa
Upgrade to lucene-8.7.0-snapshot-72d8528c3a6 (#63912)
iverase Oct 20, 2020
d9aea1c
[ML] fix inference binary classification predication label and featur…
benwtrent Oct 20, 2020
13ef3ab
Move clone snapshot API page. (#63902)
Oct 20, 2020
4bdc322
Remove two redundant DocumentMapper methods (#63922)
javanna Oct 20, 2020
c3d1981
make sure AggregationTest creates reduced aggregations. (#63931)
iverase Oct 20, 2020
ae05665
[ML] adding new flag exclude_generated that removes generated fields …
benwtrent Oct 20, 2020
2cd82f2
[Transform] add new exclude_generated flag to GET transform (#63093)
benwtrent Oct 20, 2020
8bf7976
Gradle - compatible REST test plugin - adopt bwc artifact (#63629)
jakelandis Oct 20, 2020
b4cc55c
Remove documentMapperParser method from MapperService (#63938)
javanna Oct 20, 2020
1545c0a
Minor FieldTypeLookup tweaks (#63944)
javanna Oct 20, 2020
55172b6
[Transform] add support for unsigned_long data type (#63940)
Oct 20, 2020
9a9b96b
Make agg test less confusing (#63952)
nik9000 Oct 20, 2020
7db889c
Add a test for regex usage to runtime fields (#63951)
nik9000 Oct 20, 2020
70f758e
[ML] Extend default evaluation metrics to all available (#63939)
dimitris-athanasiou Oct 21, 2020
1376afd
geo_point runtime field implementation (#63164)
iverase Oct 21, 2020
dec6ea9
Mute CoreValuesSourceTypeTests.testDatePrepareRoundingWithDocs (#63970)
javanna Oct 21, 2020
00b0bdf
Handle range query edge case (#63397)
Oct 21, 2020
473975c
Mute DatafeedJobsIT#testDatafeedTimingStats_DatafeedRecreated (#63974)
javanna Oct 21, 2020
f45ea30
Mute FieldSortIT#testCastDate (#63972)
javanna Oct 21, 2020
0482f5c
Only subject data streams to allow_auto_create, not auto_create_index
pugnascotia Oct 21, 2020
421679d
Merge remote-tracking branch 'upstream/master' into 20640-auto-create…
pugnascotia Oct 21, 2020
6bd5456
Tweak docs
pugnascotia Oct 22, 2020
0fd0f84
Merge branch 'master' into 20640-auto-create-templates
elasticmachine Oct 22, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1579,7 +1579,7 @@ public void testDataStreams() throws Exception {
CompressedXContent mappings = new CompressedXContent("{\"properties\":{\"@timestamp\":{\"type\":\"date\"}}}");
Template template = new Template(null, mappings, null);
ComposableIndexTemplate indexTemplate = new ComposableIndexTemplate(Collections.singletonList(dataStreamName), template,
Collections.emptyList(), 1L, 1L, new HashMap<>(), new ComposableIndexTemplate.DataStreamTemplate());
Collections.emptyList(), 1L, 1L, new HashMap<>(), new ComposableIndexTemplate.DataStreamTemplate(), null);
PutComposableIndexTemplateRequest putComposableIndexTemplateRequest =
new PutComposableIndexTemplateRequest().name("ds-template").create(true).indexTemplate(indexTemplate);
AcknowledgedResponse response = execute(putComposableIndexTemplateRequest,
Expand Down Expand Up @@ -1658,7 +1658,7 @@ public void testIndexTemplates() throws Exception {
Template template = new Template(settings, mappings, Map.of("alias", alias));
List<String> pattern = List.of("pattern");
ComposableIndexTemplate indexTemplate =
new ComposableIndexTemplate(pattern, template, Collections.emptyList(), 1L, 1L, new HashMap<>(), null);
new ComposableIndexTemplate(pattern, template, Collections.emptyList(), 1L, 1L, new HashMap<>(), null, null);
PutComposableIndexTemplateRequest putComposableIndexTemplateRequest =
new PutComposableIndexTemplateRequest().name(templateName).create(true).indexTemplate(indexTemplate);

Expand Down Expand Up @@ -1705,7 +1705,7 @@ public void testSimulateIndexTemplate() throws Exception {
Template template = new Template(settings, mappings, Map.of("alias", alias));
List<String> pattern = List.of("pattern");
ComposableIndexTemplate indexTemplate =
new ComposableIndexTemplate(pattern, template, Collections.emptyList(), 1L, 1L, new HashMap<>(), null);
new ComposableIndexTemplate(pattern, template, Collections.emptyList(), 1L, 1L, new HashMap<>(), null, null);
PutComposableIndexTemplateRequest putComposableIndexTemplateRequest =
new PutComposableIndexTemplateRequest().name(templateName).create(true).indexTemplate(indexTemplate);

Expand All @@ -1716,7 +1716,7 @@ public void testSimulateIndexTemplate() throws Exception {
SimulateIndexTemplateRequest simulateIndexTemplateRequest = new SimulateIndexTemplateRequest("pattern");
AliasMetadata simulationAlias = AliasMetadata.builder("simulation-alias").writeIndex(true).build();
ComposableIndexTemplate simulationTemplate = new ComposableIndexTemplate(pattern, new Template(null, null,
Map.of("simulation-alias", simulationAlias)), Collections.emptyList(), 2L, 1L, new HashMap<>(), null);
Map.of("simulation-alias", simulationAlias)), Collections.emptyList(), 2L, 1L, new HashMap<>(), null, null);
PutComposableIndexTemplateRequest newIndexTemplateReq =
new PutComposableIndexTemplateRequest().name("used-for-simulation").create(true).indexTemplate(indexTemplate);
newIndexTemplateReq.indexTemplate(simulationTemplate);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,6 @@ private static ComposableIndexTemplate randomIndexTemplate() {
if (randomBoolean()) {
dataStreamTemplate = new ComposableIndexTemplate.DataStreamTemplate();
}
return new ComposableIndexTemplate(patterns, randomTemplate(), composedOf, priority, version, meta, dataStreamTemplate);
return new ComposableIndexTemplate(patterns, randomTemplate(), composedOf, priority, version, meta, dataStreamTemplate, null);
}
}
28 changes: 19 additions & 9 deletions docs/reference/indices/put-component-template.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
<titleabbrev>Put component template</titleabbrev>
++++

Creates or updates a component template.
Component templates are building blocks for constructing <<indices-templates,index templates>>.
that specify index <<mapping,mappings>>, <<index-modules-settings,settings>>,
and <<indices-aliases,aliases>>.
Creates or updates a component template.
Component templates are building blocks for constructing <<indices-templates,index templates>>.
that specify index <<mapping,mappings>>, <<index-modules-settings,settings>>,
and <<indices-aliases,aliases>>.

[source,console]
--------------------------------------------------
Expand Down Expand Up @@ -55,10 +55,10 @@ DELETE _component_template/template_*
[[put-component-template-api-desc]]
==== {api-description-title}

An index template can be composed of multiple component templates.
An index template can be composed of multiple component templates.
To use a component template, specify it in an index template's `composed_of` list.
Component templates are only applied to new data streams and indices
as part of a matching index template.
Component templates are only applied to new data streams and indices
as part of a matching index template.

Settings and mappings specified directly in the index template or the <<indices-create-index, create index>>
request override any settings or mappings specified in a component template.
Expand Down Expand Up @@ -112,6 +112,16 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=settings]
Version number used to manage component templates externally.
This number is not automatically generated or incremented by {es}.

`allow_auto_create`::
(Optional, boolean)
This setting overrides the value of the
<<index-creation,`action.auto_create_index`>> cluster setting. If set to
`true` in a template, then indices can be automatically created using that
template even if auto-creation of indices is disabled via
`actions.auto_create_index`. If set to `false`, then indices or data streams matching the
template must always be explicitly created, and may never be automatically
created.

`_meta`::
(Optional, object)
Optional user metadata about the component template. May have any contents.
Expand Down Expand Up @@ -157,7 +167,7 @@ To be applied, a component template must be included in an index template's `com
[[component-templates-version]]
===== Component template versioning

You can use the `version` parameter to add a version number to a component template.
You can use the `version` parameter to add a version number to a component template.
External systems can use these version numbers to simplify template management.

The `version` parameter is optional and not automatically generated or used by {es}.
Expand All @@ -182,7 +192,7 @@ To check the `version`, you can use the <<getting-component-templates,get compon
[[component-templates-metadata]]
===== Component template metadata

You can use the `_meta` parameter to add arbitrary metadata to a component template.
You can use the `_meta` parameter to add arbitrary metadata to a component template.
This user-defined object is stored in the cluster state,
so keeping it short is preferrable.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.elasticsearch.http;

import org.elasticsearch.action.support.AutoCreateIndex;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.ResponseException;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.io.Streams;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.json.JsonXContent;
import org.elasticsearch.test.rest.ESRestTestCase;

import java.io.IOException;
import java.io.InputStreamReader;

import static java.nio.charset.StandardCharsets.UTF_8;
import static org.hamcrest.Matchers.containsString;

public class AutoCreateIndexIT extends ESRestTestCase {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe distribution/archives/integ-test-zip is a better place for this test class? Given that it doesn't extend HttpSmokeTestCase and this test class doesn't seem to require to start a node with special settings or plugins.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm honestly not sure. I tried before to find out where is the "correct" place for REST tests, but couldn't get a clear answer. I think I'll leave this test alone and start an email thread about it. I feel that it ought to be easier to say where a REST test should live.


/**
* Check that setting {@link AutoCreateIndex#AUTO_CREATE_INDEX_SETTING} to <code>false</code>
* disable the automatic creation on indices.
*/
public void testCannotAutoCreateIndexWhenDisabled() throws IOException {
configureAutoCreateIndex(false);

// Attempt to add a document to a non-existing index. Auto-creating the index should fail owing to the setting above.
final Request indexDocumentRequest = new Request("POST", "recipe_kr/_doc/123456");
indexDocumentRequest.setJsonEntity("{ \"name\": \"Kimchi\" }");
final ResponseException responseException = expectThrows(ResponseException.class, this::indexDocument);

assertThat(
Streams.copyToString(new InputStreamReader(responseException.getResponse().getEntity().getContent(), UTF_8)),
containsString("no such index [recipe_kr] and [action.auto_create_index] is [false]")
);
}

/**
* Check that automatically creating an index is allowed, even when {@link AutoCreateIndex#AUTO_CREATE_INDEX_SETTING}
* is <code>false</code>, when the index name matches a template and that template has <code>allow_auto_create</code>
* set to <code>true</code>.
*/
public void testCanAutoCreateIndexWhenAllowedByTemplate() throws IOException {
configureAutoCreateIndex(false);

createTemplateWithAllowAutoCreate(true);

// Attempt to add a document to a non-existing index. Auto-creating the index should succeed because the index name
// matches the template pattern
assertOK(this.indexDocument());
}

/**
* Check that automatically creating an index is disallowed when the index name matches a template and that template has
* <code>allow_auto_create</code> explicitly to <code>false</code>, even when {@link AutoCreateIndex#AUTO_CREATE_INDEX_SETTING}
* is set to <code>true</code>.
*/
public void testCannotAutoCreateIndexWhenDisallowedByTemplate() throws IOException {
configureAutoCreateIndex(true);

createTemplateWithAllowAutoCreate(false);

// Attempt to add a document to a non-existing index. Auto-creating the index should succeed because the index name
// matches the template pattern
final ResponseException responseException = expectThrows(ResponseException.class, this::indexDocument);

assertThat(
Streams.copyToString(new InputStreamReader(responseException.getResponse().getEntity().getContent(), UTF_8)),
containsString("no such index [composable template [recipe*] forbids index auto creation]")
);
}


private void configureAutoCreateIndex(boolean value) throws IOException {
XContentBuilder builder = JsonXContent.contentBuilder()
.startObject()
.startObject("transient")
.field(AutoCreateIndex.AUTO_CREATE_INDEX_SETTING.getKey(), value)
.endObject()
.endObject();

final Request settingsRequest = new Request("PUT", "_cluster/settings");
settingsRequest.setJsonEntity(Strings.toString(builder));
final Response settingsResponse = client().performRequest(settingsRequest);
assertOK(settingsResponse);
}

private void createTemplateWithAllowAutoCreate(Boolean allowAutoCreate) throws IOException {
XContentBuilder builder = JsonXContent.contentBuilder()
.startObject()
.array("index_patterns", "recipe*")
.field("allow_auto_create", allowAutoCreate)
.endObject();

final Request createTemplateRequest = new Request("PUT", "_index_template/recipe_template");
createTemplateRequest.setJsonEntity(Strings.toString(builder));
final Response createTemplateResponse = client().performRequest(createTemplateRequest);
assertOK(createTemplateResponse);
}

private Response indexDocument() throws IOException {
final Request indexDocumentRequest = new Request("POST", "recipe_kr/_doc/123456");
indexDocumentRequest.setJsonEntity("{ \"name\": \"Kimchi\" }");
return client().performRequest(indexDocumentRequest);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ public void testBulkProcessorAutoCreateRestrictions() {
assertTrue("Missing index should have been flagged", responses[1].isFailed());
assertThat(
responses[1].getFailureMessage(),
equalTo("[wontwork] org.elasticsearch.index.IndexNotFoundException: no such index [wontwork]"));
equalTo("[wontwork] org.elasticsearch.index.IndexNotFoundException: no such index [wontwork]"
+ " and [action.auto_create_index] is [false]"));
assertFalse("Operation on existing index should succeed", responses[2].isFailed());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ public void testUsageOfDataStreamFails() throws IOException {
Exception expectedException = expectThrows(Exception.class, () -> ComposableIndexTemplate.parse(parser));

ComposableIndexTemplate template = new ComposableIndexTemplate(List.of("logs-*-*"), null, null, null, null,
null, new ComposableIndexTemplate.DataStreamTemplate());
null, new ComposableIndexTemplate.DataStreamTemplate(), null);
Exception e = expectThrows(IllegalArgumentException.class, () -> client().execute(PutComposableIndexTemplateAction.INSTANCE,
new PutComposableIndexTemplateAction.Request("my-it").indexTemplate(template)).actionGet());
Exception actualException = (Exception) e.getCause();
Expand Down
Loading