Ilm alias #3826
Merged
Ilm alias #3826
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* rename default to unmanaged indices * move index selector config to ilm and unmanaged packages * ilm: replace variable index part when loading config * ensure index and template names are correctly resolved
Codecov Report
@@ Coverage Diff @@
## master #3826 +/- ##
==========================================
- Coverage 79.92% 79.64% -0.28%
==========================================
Files 137 135 -2
Lines 6166 6142 -24
==========================================
- Hits 4928 4892 -36
- Misses 1238 1250 +12
|
❕ Build Aborted
Expand to view the summary
Build stats
Log outputExpand to view the last 100 lines of log output
|
bmorelli25
reviewed
May 27, 2020
There was a problem hiding this comment.
Docs are looking good. A few more changes need to be made:
- The ILM reference docs (
docs/ilm-reference.asciidoc):- I think we should update the "Default configuration" section to include
rollover_alias:
mapping: - event_type: "error" policy_name: "apm-rollover-30-days" rollover_alias: "apm-%{[observer.version]}-error" - event_type: "span" policy_name: "apm-rollover-30-days" rollover_alias: "apm-%{[observer.version]}-span" - event_type: "transaction" policy_name: "apm-rollover-30-days" rollover_alias: "apm-%{[observer.version]}-transaction" - event_type: "metric" policy_name: "apm-rollover-30-days" rollover_alias: "apm-%{[observer.version]}-metric"- We also need to update the configuration reference entry for
apm-server.ilm.setup.mappingat the bottom of the page. Something like this:
[float] [[ilm-setup-mapping-config]] ===== `apm-server.ilm.setup.mapping` Maps each event type to a corresponding `policy_name` and `rollover_alias`. APM event types can only be `error`, `span`, `transaction`, and `metric`. If you attempt to map an index lifecycle policy to a different event type, APM Server will not start. If you only map a subset of APM event types, the default values will be used for omitted event types. If a policy is defined, it must be mapped to an event type; if it isn't, it will not be sent to Elasticsearch. When customizing `rollover_alias`, it is strongly recommended to keep the prefix `apm-%{[observer.version]}` unchanged; if the prefix must be changed, `setup.template.name` and `setup.template.pattern` need to be changed accordingly, to match the created indices. - I think we should update the "Default configuration" section to include
bmorelli25
reviewed
May 27, 2020
axw
reviewed
May 28, 2020
* create common package with index defaults * improve `assert_alias` in system tests * minor test improvements
axw
approved these changes
May 28, 2020
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
bmorelli25
approved these changes
May 28, 2020
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Jun 10, 2020
Change implementation to only allow configuring an `apm-server.ilm.setup.mapping.*.index_suffix` rather than the whole rollover alias. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Jun 18, 2020
Change how rollover aliases can be configured to only allow customizingi a suffix.This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Jun 18, 2020
Change how rollover aliases can be configured to only allow customizingi a suffix.This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826 closes elastic#3895
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Jun 18, 2020
Change how rollover aliases can be configured to only allow customizingi a suffix.This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826 closes elastic#3895
4 tasks
simitt
added a commit
that referenced
this pull request
Jun 23, 2020
Change how rollover aliases can be configured to only allow customizing a suffix. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on #3826 closes #3895 Co-authored-by: Brandon Morelli <[email protected]>
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Jun 23, 2020
* Refactor ILM config loading and conditional index selection * Make rollover_alias configurable and document it closes elastic#3779
simitt
added a commit
that referenced
this pull request
Jun 23, 2020
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Jun 23, 2020
Change how rollover aliases can be configured to only allow customizing a suffix. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826 closes elastic#3895 Co-authored-by: Brandon Morelli <[email protected]>
simitt
added a commit
that referenced
this pull request
Jun 23, 2020
Change how rollover aliases can be configured to only allow customizing a suffix. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on #3826 closes #3895 Co-authored-by: Brandon Morelli <[email protected]>
|
Replaced by #3905, removing this from test plan |
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Oct 27, 2020
When configuring `ilm.setup.require_policy:false` the customized policy name has been ignored. This fixes the issue by ensuring the policy name is added to the ILM setup. The regression was introduced with elastic#3826 fixes 4349
2 tasks
simitt
added a commit
that referenced
this pull request
Oct 27, 2020
When configuring `ilm.setup.require_policy:false` the customized policy name has been ignored. This fixes the issue by ensuring the policy name is added to the ILM setup. The regression was introduced with #3826 fixes 4349
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Oct 27, 2020
When configuring `ilm.setup.require_policy:false` the customized policy name has been ignored. This fixes the issue by ensuring the policy name is added to the ILM setup. The regression was introduced with elastic#3826 fixes 4349
simitt
added a commit
to simitt/apm-server
that referenced
this pull request
Oct 27, 2020
When configuring `ilm.setup.require_policy:false` the customized policy name has been ignored. This fixes the issue by ensuring the policy name is added to the ILM setup. The regression was introduced with elastic#3826 fixes 4349
simitt
added a commit
that referenced
this pull request
Oct 27, 2020
simitt
added a commit
that referenced
this pull request
Oct 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation/summary
With this PR we allow users to customize the ILM
rollover_aliasfor every event type.It also improves the ILM specific config handling and separates code for unmanaged indices into a dedicated package for easier distinction between setup for ILM and unmanaged (old) indices.
Docs Update
Users need to be aware that whenever they switch to a different prefix than
apm-version*they also need to modifysetup.template.nameandsetup.template.patternas otherwise the generic index template will not match the event indices. This is similar to what's already true when users configureoutput.elasticsearch.indices.@bmorelli25 I'd appreciate your eyes on the docs changes I made.
Checklist
-
[ ] I have signed the Contributor License Agreement.I have considered changes for:
-
[ ] logging (add log lines, choose appropriate log selector, etc.)-
[ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)-
[ ] telemetry-
[ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)-
[ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)How to test these changes
Configure different settings for
apm-server.ilm.setup.mappingand check that event aliases, policies, indices and templates get set up accordingly. The event specific templates must match against the customized indices and reference the aliases correctly.Related issues
#3779