[BUG] Cluster State API is undocumented and unavailable in clients

[dbwiddis]

Describe the bug

Coming from opensearch-project/OpenSearch#7066 (comment)

Internally to OpenSearch, the cluster state is retrieved by handling the ClusterStateAction / TransportClusterStateAction / ClusterStateRequest / ClusterStateResponse implementations. The RestClusterStateAction exposes the API (?) as:

• /_cluster/state
• /_cluster/state/{metric}
• /_cluster/state/{metric}/{indices}

However, there is no (0 / zero / null) documentation of this API on the OpenSearch website. Entering "/_cluster/state" in the search bar at https://opensearch.org/ gives 17 responses, out of which zero contain the text "/_cluster/state".

Entering "/_cluster/state" and "OpenSearch" as key words into your favorite search engine give no results on https://opensearch.org/ (other than one of the previous 17 that doesn't actually include that text). There are some third-party references to the API, a reference to the API at ElasticSearch, and inclusion as a "common" API request on AOS documentation page.

More importantly, this "API" is not accessible via OpenSearch clients. Clients are spec-driven, but the Cluster APIs do not include it. While it's possible to reverse-engineer the ClusterStateRequest to construct a raw endpoint for performRequest() on the clients, the returned response is just JSON and parsing it into a ClusterStateResponse is an exercise in futility.

Expected behavior

1. The API is documented on the OpenSearch website. (Definitely.)
2. The ClusterStateRequest/ClusterStateResponse objects are usable in OpenSearch clients in all languages. (Maybe not, see comments below.)

[dbwiddis]

But, I wonder if we want to expose all _cluster/state API constructs to clients. I foresee lot of challenges:

1. To start with "custom" that you mentioned above. Plugins can add any custom data in cluster state and it would be breaking change if fromXContent is enforced. Also, plugins can choose not to send certain data in API response and mark it private.
2. Today, these internal data structures are updated just thinking about server side backward compatibility. Now changing anything in those structures could potentially break clients and would result in too much overhead.
3. If you look at the toXContent method of some of the classes you will notice, not all fields are serialized, only a subset which might not be sufficient to deserialize the object back to current ClusterState data structures. e.g. anywhere Index is serialized in routing information, only its name is serialized not Index UUID. Metadata classes have fromXContent already implemented as it is used to serde to and from disk as well.

I see this API more of helping in debugging and providing it as official Cluster APIs in client could cause lot of maintenance overhead. Until community feels the need with real use cases, we should refrain from exposing it.

Originally posted by @shwetathareja in opensearch-project/OpenSearch#7066 (comment)

[dblock]

Should we move this to documentation-website as a documentation bug?

[minalsha]

moving it to documentation-website repo as a documentation bug

[dbwiddis]

Should we move this to documentation-website as a documentation bug?

@dblock this should probably be either moved to the api repo now, or closed if it's actually already been completed?