Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Cluster State Stats Fails For Multi Node Setup #10961

Closed
amkhar opened this issue Oct 27, 2023 · 1 comment · Fixed by #10964
Closed

[BUG] Cluster State Stats Fails For Multi Node Setup #10961

amkhar opened this issue Oct 27, 2023 · 1 comment · Fixed by #10964
Labels
bug Something isn't working untriaged

Comments

@amkhar
Copy link
Contributor

amkhar commented Oct 27, 2023

Describe the bug
curl localhost:9200/_nodes/stats?pretty fails with following exception

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Field name cannot be null"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Field name cannot be null",
    "suppressed" : [
      {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unclosed object or array found"
        }
      }
    ]
  },
  "status" : 400
}

To Reproduce
Steps to reproduce the behavior:

  1. Create a multi node cluster
  2. Run _nodes/stats

Expected behavior
_nodes/stats should always work fine for single node or multi node clusters

Additional context
Recently cluster state stats were added #10670 which is causing this issue.

@amkhar amkhar added bug Something isn't working untriaged labels Oct 27, 2023
@amkhar
Copy link
Contributor Author

amkhar commented Oct 27, 2023

Root cause : It was happening as statsName field is not being written on stream and while creating PersistedStateStats object it's not being set.

Resolution : Write statsName field on stream and read it to set it in the object.

Stack trace

[2023-10-27T15:57:11,821][INFO ][o.o.c.c.PersistedStateStats] [clusterManager2] stats name null
[2023-10-27T15:57:11,821][WARN ][stderr                   ] [clusterManager2] java.lang.IllegalArgumentException: Field name cannot be null
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.core.xcontent.XContentBuilder.ensureNotNull(XContentBuilder.java:1058)
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.core.xcontent.XContentBuilder.ensureNameNotNull(XContentBuilder.java:1053)
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.core.xcontent.XContentBuilder.field(XContentBuilder.java:314)
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.core.xcontent.XContentBuilder.startObject(XContentBuilder.java:291)
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.cluster.coordination.PersistedStateStats.toXContent(PersistedStateStats.java:76)
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.cluster.service.ClusterStateStats.toXContent(ClusterStateStats.java:107)
[2023-10-27T15:57:11,822][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.discovery.DiscoveryStats.toXContent(DiscoveryStats.java:93)
[2023-10-27T15:57:11,823][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.action.admin.cluster.node.stats.NodeStats.toXContent(NodeStats.java:545)
[2023-10-27T15:57:11,823][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.action.admin.cluster.node.stats.NodesStatsResponse.toXContent(NodesStatsResponse.java:78)
[2023-10-27T15:57:11,823][WARN ][stderr                   ] [clusterManager2] 	at org.opensearch.rest.action.RestActions.nodesResponse(RestActions.java:223)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working untriaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant