_all and * resolves to system indices and causes WARN deprecation logs

[rudolf]

Examples (affects 7.16 and main):

curl -IX GET "elastic:changeme@localhost:9200/_all/_rollup/data" -H 'Content-Type: application/json' 
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
Warning: 299 Elasticsearch-7.16.0-SNAPSHOT-34fb605e5a51f9ecbf4a6ad3d3b910402c5b2121 "this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default"
content-type: application/json; charset=UTF-8
content-length: 2

kibana % curl -IX GET "http://elastic:changeme@localhost:9200/*/_search?pretty" -H "x-elastic-product-origin: kibana"
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
Warning: 299 Elasticsearch-8.1.0-SNAPSHOT-838597e18e0495a2239a71187df0b49b69c17f77 "this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default"
content-type: application/json
content-length: 2385

I couldn't find any existing issues but found the following related issues/comments:
#50251 (comment)
#78850

Related: elastic/kibana#120043

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[williamrandolph]

I merged a PR into master that changes this behavior there: #79512

The behavior is intended in the 7.x line.

[rudolf]

For Kibana we will hide this deprecation warning from users by specifying the x-elastic-product-origin: kibana header.

But stepping back and thinking about other users that integrate directly with Elasticsearch. If they use _all or * I don't see a way for them to resolve this deprecation message in 7.x so that they can gain confidence to upgrade?

[williamrandolph]

Our guidance would be to change their query pattern to avoid using those wildcard queries. At a bare minimum, I believe they could use the pattern *,-.security-7 to avoid the warning. If they have a use case where they really need to access system resources, we would like to know about it so we can provide that access through a plugin.

We already downgraded this warning from CRITICAL to WARN. I think it's an important warning. We really do hope to be able to remove all direct access to these indices some day, but we just couldn't get there in time for 8.0.

[rudolf]

@williamrandolph thanks, that makes sense. I think the challenge for our users is how to track down the source of this deprecation warning and it isn't obvious that a * wildcard is the reason they're triggering a deprecation warning. Also .security-7 isn't the only system index, so they would need to use something like *,-.* which would also exclude hidden indices which might or might not be what they want.

this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default

If we could change this message to something like the following I think it would help users a lot in tracking down the problem query: "This request uses an index pattern [*] that expands to include system indices. In a future major version wildcards will no longer expand to match system indices." Ideally we'd have something actionable like "To prevent this deprecation warning..." too.

[gwbrown]

If we could change this message to something like the following

I agree that it would be very helpful, but unfortunately due to how Security is designed we don't reliably have access to the index patterns the user supplied at that time - they're basically overwritten by Security with expanded versions that are guaranteed to only fall within the indices that user has permissions to access. That means that the "pattern" that matched that index at the time we emit the warning is very likely different from what the user supplied.

We actually investigated this when we first implemented the warning, explicitly to special case _all/*, and while we ended up not going that route for other reasons, it was quite painful to communicate all the necessary information from Security to Elasticsearch core. We've talked about architectural changes that would allow us to resolve this, but won't have bandwidth to implement them any time in the immediate future.

It's worth noting that while this is an issue in 7.x, this typically won't be an issue post-8.0, as system indices will be restricted to users with allow_restricted_indices permissions, which most users will not have.

Given that we won't really have bandwidth to address this for the next 7.x release, and the reduced impact in 8.0, I'm going to close this, at least for now. Once we decide the long-term path for system indices (specifically, to what degree we're going to lock them down above what's in 8.0 right now) we'll likely need to reevaluate some things here anyway.