GET _aliases doesn't respect x-elastic-product-origin header

[sabarasaba]

Elasticsearch version 8.1

Kibana version: 8.1

While working on elastic/kibana#121857 we found out that querying GET _aliases doesn't respect the x-elastic-product-origin=kibana and generates deprecation logs about accessing the .security, .security-7 and .tasks system indices.

JSON response without header

#! this request accesses system indices: [.apm-agent-configuration, .kibana_security_session_1, .fleet-policies-7, .kibana_8.1.0_001, .apm-custom-link, .tasks, .fleet-enrollment-api-keys-7, .kibana_task_manager_8.1.0_001, .security-7], but in a future major version, direct access to system indices will be prevented by default
#! this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .security, .fleet-enrollment-api-keys, .kibana, .kibana_8.1.0, .fleet-policies, .kibana_task_manager_8.1.0], but in a future major version, direct access to system indices and their aliases will not be allowed
{
  ".kibana-event-log-8.1.0-000001" : {
    "aliases" : {
      ".kibana-event-log-8.1.0" : {
        "is_write_index" : true,
        "is_hidden" : true
      }
    }
  },
  ".apm-agent-configuration" : {
    "aliases" : { }
  },
  ".kibana_security_session_1" : {
    "aliases" : { }
  },
  "kibana_sample_data_flights" : {
    "aliases" : { }
  },
  ".fleet-policies-7" : {
    "aliases" : {
      ".fleet-policies" : {
        "is_hidden" : true
      }
    }
  },
  ".kibana_8.1.0_001" : {
    "aliases" : {
      ".kibana" : {
        "is_hidden" : true
      },
      ".kibana_8.1.0" : {
        "is_hidden" : true
      }
    }
  },
  ".apm-custom-link" : {
    "aliases" : { }
  },
  ".tasks" : {
    "aliases" : { }
  },
  ".fleet-enrollment-api-keys-7" : {
    "aliases" : {
      ".fleet-enrollment-api-keys" : {
        "is_hidden" : true
      }
    }
  },
  ".kibana_task_manager_8.1.0_001" : {
    "aliases" : {
      ".kibana_task_manager" : {
        "is_hidden" : true
      },
      ".kibana_task_manager_8.1.0" : {
        "is_hidden" : true
      }
    }
  },
  ".security-7" : {
    "aliases" : {
      ".security" : {
        "is_hidden" : true
      }
    }
  }
}

JSON response with header

#! this request accesses system indices: [.tasks, .security-7], but in a future major version, direct access to system indices will be prevented by default
#! this request accesses aliases with names reserved for system indices: [.security], but in a future major version, direct access to system indices and their aliases will not be allowed
{
  ".fleet-policies-7" : {
    "aliases" : {
      ".fleet-policies" : {
        "is_hidden" : true
      }
    }
  },
  ".kibana_task_manager_8.1.0_001" : {
    "aliases" : {
      ".kibana_task_manager" : {
        "is_hidden" : true
      },
      ".kibana_task_manager_8.1.0" : {
        "is_hidden" : true
      }
    }
  },
  ".kibana_security_session_1" : {
    "aliases" : { }
  },
  ".apm-agent-configuration" : {
    "aliases" : { }
  },
  ".tasks" : {
    "aliases" : { }
  },
  ".kibana_8.1.0_001" : {
    "aliases" : {
      ".kibana" : {
        "is_hidden" : true
      },
      ".kibana_8.1.0" : {
        "is_hidden" : true
      }
    }
  },
  ".security-7" : {
    "aliases" : {
      ".security" : {
        "is_hidden" : true
      }
    }
  },
  ".kibana-event-log-8.1.0-000001" : {
    "aliases" : {
      ".kibana-event-log-8.1.0" : {
        "is_write_index" : true,
        "is_hidden" : true
      }
    }
  },
  ".apm-custom-link" : {
    "aliases" : { }
  },
  "kibana_sample_data_flights" : {
    "aliases" : { }
  },
  ".fleet-enrollment-api-keys-7" : {
    "aliases" : {
      ".fleet-enrollment-api-keys" : {
        "is_hidden" : true
      }
    }
  }
}

[cjcenizal]

@ywangd Is this related to #81589 ?

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ywangd]

No I think they are different. This issue is that ES does not honor the x-elastic-product-origin header when issue deprecation warning for accessing system indices. That is, the system indices are accessed, but ES should not issue warnings because of the header.

The other one is that the system indices should not be considered as "accessed" at all because they do not match the requested index pattern.

Both should not generate deprecation warnings but for different reasons. I hope this makes sense.

[gwbrown]

Okay, having had some time to dig into this, I think there's some expectations that aren't aligned here.

The behavior here is intentional: When Kibana makes this request, it runs into #81345, where accessing */_all causes deprecation warnings due to system index access. When the header isn't present, the list of indices causing the warning is just all of the system indices in the cluster. When the header is present, we allow access to the system indices owned by that product origin (Kibana), but the request still accesses system indices that Kibana doesn't own - .tasks and .security-7 in this case.

Now, my understanding was that we were going to rely on #81381 in order to not show users deprecation logs that aren't relevant to them, rather than disabling emitting deprecation logs entirely for valid product-origin headers.

Does that make sense @cjcenizal / @sabarasaba? I'm fine changing Elasticsearch's behavior if I've missed something, but per my understanding there's not a bug here.

[cjcenizal]

@gwbrown Thanks for explaining. I think you're right that there was just a misunderstanding about intention and desired behavior. To try to clarify things, the end goal is to keep Upgrade Assistant free of any logs that have been triggered by requests that are beyond the user's control. We want to use the Kibana product origin to signal to Elasticsearch that the request is "beyond the user's control" and therefore all logs that are subsequently emitted as a result of the request need to be marked so they can be hidden from the user. Does this make sense?

I think we're on the same page regarding #81381. We implemented elastic/kibana#121389 and elastic/kibana#121174 to filter out any logs that originated from an Elastic product. I have a question regarding the situation you're describing:

the request still accesses system indices that Kibana doesn't own - .tasks and .security-7 in this case.

Is an elastic_product_origin field attached to the logs that are emitted as a result? If so, it seems like there might be a bug on our end. What's the value of that field?

[rudolf]

In my tests we aren't propagating the product origin to the deprecation log:

rudolf@rudolf-elastic-mbp kibana-7.16 % curl -v -X GET "elastic:changeme@localhost:9200/_aliases" -H 'x-elastic-product-origin: kibana'                          
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
* Server auth using Basic with user 'elastic'
> GET /_aliases HTTP/1.1
> Host: localhost:9200
> Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==
> User-Agent: curl/7.64.1
> Accept: */*
> x-elastic-product-origin: kibana
> 
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< Warning: 299 Elasticsearch-7.17.0-SNAPSHOT-677ac9fa5806ca27e905479aec78ca06ca36e973 "this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default"
< Warning: 299 Elasticsearch-7.17.0-SNAPSHOT-677ac9fa5806ca27e905479aec78ca06ca36e973 "this request accesses aliases with names reserved for system indices: [.security], but in a future major version, direct access to system indices and their aliases will not be allowed"
< content-type: application/json; charset=UTF-8
< content-length: 44
< 
* Connection #0 to host localhost left intact
{".security-7":{"aliases":{".security":{}}}}
* Closing connection 0

rudolf@rudolf-elastic-mbp kibana-7.16 % curl -X POST "elastic:changeme@localhost:9200/.logs-deprecation*/_search?pretty=true" -H 'Content-Type: application/json'
{
  "took" : 16,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-.logs-deprecation.elasticsearch-default-2022.01.24-000001",
        "_type" : "_doc",
        "_id" : "nKfEjX4BI8-iZEj1dP1R",
        "_score" : 1.0,
        "_source" : {
          "event.dataset" : "deprecation.elasticsearch",
          "@timestamp" : "2022-01-24T21:26:04,907+01:00",
          "log.level" : "WARN",
          "log.logger" : "org.elasticsearch.deprecation.action.admin.indices.alias.get.TransportGetAliasesAction",
          "elasticsearch.cluster.name" : "elasticsearch",
          "elasticsearch.cluster.uuid" : "4foBwb94Re6X5O7FA11tMA",
          "elasticsearch.node.id" : "Kpfc--4IQD-SIWMeYWF_XQ",
          "elasticsearch.node.name" : "rudolf-elastic-mbp.home",
          "trace.id" : "",
          "message" : "this request accesses system indices: [.security-7], but in a future major version, direct access to system indices will be prevented by default",
          "data_stream.type" : "logs",
          "data_stream.dataset" : "deprecation.elasticsearch",
          "data_stream.namespace" : "default",
          "ecs.version" : "1.7",
          "elasticsearch.event.category" : "api",
          "event.code" : "open_system_index_access",
          "elasticsearch.http.request.x_opaque_id" : ""
        }
      },
      {
        "_index" : ".ds-.logs-deprecation.elasticsearch-default-2022.01.24-000001",
        "_type" : "_doc",
        "_id" : "nafEjX4BI8-iZEj1dP1T",
        "_score" : 1.0,
        "_source" : {
          "event.dataset" : "deprecation.elasticsearch",
          "@timestamp" : "2022-01-24T21:26:04,913+01:00",
          "log.level" : "WARN",
          "log.logger" : "org.elasticsearch.deprecation.action.admin.indices.alias.get.TransportGetAliasesAction",
          "elasticsearch.cluster.name" : "elasticsearch",
          "elasticsearch.cluster.uuid" : "4foBwb94Re6X5O7FA11tMA",
          "elasticsearch.node.id" : "Kpfc--4IQD-SIWMeYWF_XQ",
          "elasticsearch.node.name" : "rudolf-elastic-mbp.home",
          "trace.id" : "",
          "message" : "this request accesses aliases with names reserved for system indices: [.security], but in a future major version, direct access to system indices and their aliases will not be allowed",
          "data_stream.type" : "logs",
          "data_stream.dataset" : "deprecation.elasticsearch",
          "data_stream.namespace" : "default",
          "ecs.version" : "1.7",
          "elasticsearch.event.category" : "api",
          "event.code" : "open_system_alias_access",
          "elasticsearch.http.request.x_opaque_id" : ""
        }
      }
    ]
  }
}

Similarly some other deprecated API calls also don't propagate it:

curl -v -X PUT "elastic:changeme@localhost:9200/.testtype?include_type_name=true&pretty=true" -H 'x-elastic-product-origin: kibana'

        "_source" : {
          "event.dataset" : "deprecation.elasticsearch",
          "@timestamp" : "2022-01-24T21:28:17,992+01:00",
          "log.level" : "CRITICAL",
          "log.logger" : "org.elasticsearch.deprecation.rest.action.admin.indices.RestCreateIndexAction",
          "elasticsearch.cluster.name" : "elasticsearch",
          "elasticsearch.cluster.uuid" : "4foBwb94Re6X5O7FA11tMA",
          "elasticsearch.node.id" : "Kpfc--4IQD-SIWMeYWF_XQ",
          "elasticsearch.node.name" : "rudolf-elastic-mbp.home",
          "trace.id" : "",
          "message" : "[types removal] Using include_type_name in create index requests is deprecated. The parameter will be removed in the next major version.",
          "data_stream.type" : "logs",
          "data_stream.dataset" : "deprecation.elasticsearch",
          "data_stream.namespace" : "default",
          "ecs.version" : "1.7",
          "elasticsearch.event.category" : "types",
          "event.code" : "create_index_with_types",
          "elasticsearch.http.request.x_opaque_id" : ""
        }
      }

[gwbrown]

@rudolf This turned out not to be what I expected! There appears to be a bug in 7.17 (only, not in master) that causes the elasticsearch.elastic_product_origin field to only be logged if there's an X-Opaque-Id present, which appears to be unintentional. I've opened #83115 to fix that behavior, and with that fix both examples include the origin in the JSON log.

[gwbrown]

This should be fixed as #83115 was merged a while ago. Please reopen the issue if we see this again.