Add an API to remove System Indices

[gwbrown]

As part of formalizing System Indices in Elasticsearch (#50251), we will be restricting access to System Indices via the REST interface, except by means of purpose-built APIs. As has been raised in #62501, this will require a new API to remove System Indices, for the purposes of cleanup between tests if nothing else. We also believe that this will be of use to end users, who may wish to "reset" features which store configuration or other information in system indices.

Simply deleting all System Indices is not sufficient to accomplish this goal without causing other problems. Some features store information for "in flight" tasks in their associated System Indices, for example. Because different features may have different requirements, it may be desirable to add several APIs, rather than a single one.

One way to implement this would be to simply have each plugin add a _reset (or similar) API endpoint which performs whatever tasks are required to clean that plugin's indices/tasks/etc.

Another way would be to add a resetSystemIndices() method to the SystemIndexPlugin interface, which is required to clean up all System Indices owned by that plugin, as well as any other tasks (or provide a callback to do so, etc.). This could be called from a central "factory reset" API.

The two options are not mutually exclusive, either. Whatever the implementation, this reset logic should be owned by the team which owns the plugin in question, in order to ensure that the cleanup is complete and kept up to date with changes to the plugin.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[gwbrown]

Continuing discussion from this comment

I think a "factory reset" API should be added as a must to the meta issue before eventually locking the access to system indices.

Agreed. I've added this issue to the meta issue, and it's something we'll definitely have to address before fully locking down REST access to System Indices - otherwise, we'd see massive numbers of test failures due to test cases not starting from a "clean slate" as they were designed for.

That said, we can move forward with deprecation of REST access to System Indices prior to that - if necessary, we can simply ignore the deprecation warning during test cleanup, as long there's a plan to address the issue.

We see users doing a manual reset by deleting the indices very often in ML. A challenge to that: For a factory reset any jobs(ml/transform/rollup/...) must be stopped.

This is great info - and something I think supports the approach of having a (or multiple) "reset" APIs. The goal of the System Indices project as a whole is to make it less likely that users will cause problems in their cluster by accessing indices that are "implementation details" of certain features. Users causing problems by manually deleting the ML indices, which only partially accomplishes their goal, is a very good example of the issues we're trying to solve.

[gwbrown]

Another possibility that's been brought up is creating a "test plugin" which exposes an API that emulates the behavior of DELETE _all today. This would not bring along the user-facing benefits of a "factory reset" API, but would allow us to move forward with our plans for System Indices without having to immediately coordinate everything involved in productizing this sort of "reset" while still allowing the development of such an API in the near future.

However, it is not currently clear if this approach would be viable for the Clients tests. I will be reaching out to the Clients team to ask if using such a plugin at test time would be viable for their use case, and update this issue with the conclusion.

[droberts195]

However, it is not currently clear if this approach would be viable for the Clients tests. I will be reaching out to the Clients team to ask if using such a plugin at test time would be viable for their use case, and update this issue with the conclusion.

It was just pointed out to me that the Kibana functional tests for ML also need to delete system indices in between tests. I imagine the same might be true for some of the solutions that are Kibana plugins but access system indices. So we'll need to make sure that the solution to this problem is accessible from the Kibana functional test framework as well as client test frameworks.