Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce standalone data stream support, deprecate Elasticsearch setup #5914

Closed
axw opened this issue Aug 10, 2021 · 1 comment · Fixed by #6145
Closed

Introduce standalone data stream support, deprecate Elasticsearch setup #5914

axw opened this issue Aug 10, 2021 · 1 comment · Fixed by #6145
Assignees
Milestone

Comments

@axw
Copy link
Member

axw commented Aug 10, 2021

APM Server is currently responsible for installing Elasticsearch (legacy) index templates, ingest pipelines, and ILM policies. We now also have the Fleet integration which installs variants of these, with the index templates creating traces-*, logs-*, and metrics-* data streams instead of apm-* indices.

We intend to increasingly rely on Fleet and Elastic Agent for powering features such as APM agent central config and source map distribution. In the future, we will remove the ability to write to apm-* indices and always create data streams.

We should introduce an option for standalone APM Server to start writing data streams, relying on Fleet (Kibana) to set up the integration package. When this option is enabled, APM Server will wait for the integration package to be installed before writing any data, and will write to the data stream names rather than legacy index names.

I propose that we introduce the option no later than 7.last, where it will be off by default. At the same time we will deprecate Elasticsearch setup.
In 8.0 we will remove the ability to setup Elasticsearch, and turn the option on by default (#5961). It should still be possible to disable the option for scenarios where APM Server lacks privileges to check the package is installed; in this case the user is responsible for ensuring the package is installed before APM Server is started.

@axw
Copy link
Member Author

axw commented Sep 6, 2021

Related: #5949

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants