[BUG] org.opensearch.rest.action.admin.cluster.RestNodesStatsActionTests.testIndexMetricsRequestWithoutIndicesMetric is flaky

[rahulkarajgikar]

Describe the bug

Error Message

java.lang.AssertionError: 
Expected: with toString() a string containing "request [/_nodes/stats] contains index metrics [store] but indices stats not requested"
     but: toString() was "java.lang.IllegalArgumentException: request [/_nodes/stats] contains unrecognized cache type: [store]"

Stack Trace

java.lang.AssertionError: 
Expected: with toString() a string containing "request [/_nodes/stats] contains index metrics [store] but indices stats not requested"
     but: toString() was "java.lang.IllegalArgumentException: request [/_nodes/stats] contains unrecognized cache type: [store]"
	at __randomizedtesting.SeedInfo.seed([59C34146379C7823:E2D005FBBA7D0E0B]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.junit.Assert.assertThat(Assert.java:930)
	at org.opensearch.rest.action.admin.cluster.RestNodesStatsActionTests.testIndexMetricsRequestWithoutIndicesMetric(RestNodesStatsActionTests.java:145)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Standard Output:

[2024-05-01T00:59:14,081][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testAllRequestWithOtherMetrics] before test
[2024-05-01T00:59:14,210][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testAllRequestWithOtherMetrics] after test
[2024-05-01T00:59:14,226][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedIndexMetric] before test
[2024-05-01T00:59:14,228][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedIndexMetric] after test
[2024-05-01T00:59:14,241][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testIndexMetricsRequestOnAllRequest] before test
[2024-05-01T00:59:14,242][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testIndexMetricsRequestOnAllRequest] after test
[2024-05-01T00:59:14,256][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedMetricDidYouMean] before test
[2024-05-01T00:59:14,257][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedMetricDidYouMean] after test
[2024-05-01T00:59:14,271][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedMetric] before test
[2024-05-01T00:59:14,273][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedMetric] after test
[2024-05-01T00:59:14,289][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedIndexMetricDidYouMean] before test
[2024-05-01T00:59:14,290][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testUnrecognizedIndexMetricDidYouMean] after test
[2024-05-01T00:59:14,304][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testIndexMetricsRequestWithoutIndicesMetric] before test
[2024-05-01T00:59:14,311][INFO ][o.o.r.a.a.c.RestNodesStatsActionTests] [testIndexMetricsRequestWithoutIndicesMetric] after test

Standard Error:

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.rest.action.admin.cluster.RestNodesStatsActionTests.testIndexMetricsRequestWithoutIndicesMetric" -Dtests.seed=59C34146379C7823 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PE -Dtests.timezone=Asia/Dubai -Druntime.java=21
NOTE: leaving temporary files on disk at: /var/jenkins/workspace/gradle-check/search/server/build/testrun/test/temp/org.opensearch.rest.action.admin.cluster.RestNodesStatsActionTests_59C34146379C7823-001
NOTE: test params are: codec=Asserting(Lucene99): {}, docValues:{}, maxPointsInLeafNode=885, maxMBSortInHeap=6.381069237088775, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=es-PE, timezone=Asia/Dubai
NOTE: Linux 5.15.0-1056-aws amd64/Eclipse Adoptium 21.0.3 (64-bit)/cpus=32,threads=1,free=250443016,total=536870912
NOTE: All tests run in this JVM: [DocWriteResponseTests, TransportAddVotingConfigExclusionsActionTests, CancellableTasksTests, ClusterGetSettingsResponseTests, SnapshotStatusTests, ScriptParameterInfoSerializingTests, CreateIndexResponseTests, GetMappingsResponseTests, IndicesSegmentsRequestTests, GetIndexTemplatesResponseTests, BulkPrimaryExecutionContextTests, ExplainResponseTests, PutPipelineRequestTests, ClearScrollControllerTests, SearchPhaseExecutionExceptionTests, TransportDeletePitActionTests, ShardsAcknowledgedResponseTests, TermVectorsUnitTests, MultiCommandTests, ShardFailedClusterStateTaskExecutorTests, CoordinationCheckerSettingsTests, OpenSearchNodeCommandTests, ComponentTemplateMetadataTests, IndexNameExpressionResolverAliasIterationTests, ViewTests, PlainShardIteratorTests, AllocationPriorityTests, FailedShardsRoutingTests, RebalanceAfterActiveTests, ThrottlingAllocationTests, DiffableTests, PidFileTests, ByteBuffersBytesReferenceTests, CopyOnWriteHashMapTests, GeometryIndexerTests, MurmurHash3Tests, ShardCoreKeyMapTests, RegexTests, BoundTransportAddressTests, LongHashTests, CountDownTests, IterablesTests, CborXContentFilteringTests, InitializeExtensionRequestTests, ExtensionActionResponseTests, DanglingIndicesStateTests, ClusterMetadataManifestTests, OpenSearchTieredMergePolicyTests, HunspellTokenFilterFactoryTests, EngineConfigTests, FieldDataCacheTests, FieldStatsProviderRefreshTests, DefaultAnalyzersTests, FieldAliasMapperValidationTests, IndexFieldMapperTests, MultiFieldCopyToMapperTests, RootObjectMapperTests, CommonTermsQueryBuilderTests, GeoShapeQueryBuilderGeoShapeTests, PrefixIntervalsSourceProviderTests, SimpleQueryStringBuilderTests, WrapperQueryBuilderTests, RemoteIndexPathTests, SimpleQueryStringQueryParserTests, RetentionLeaseStatsTests, NewPathForShardTests, ShardUtilsTests, RemoteSegmentStoreDirectoryFactoryTests, CompletionsStatsTests, IRCKeyWriteableSerializerTests, BreakerSettingsTests, RemoteStoreReplicationSourceTests, IngestServiceTests, FloatRandomBinaryDocValuesRangeQueryTests, JvmInfoTests, CancelPersistentTaskRequestTests, PluginsTests, IndexMetadataGenerationsTests, RestHttpResponseHeadersTests, RestNodesStatsActionTests]

Related component

Indexing

To Reproduce

CI - https://build.ci.opensearch.org/job/gradle-check/38104/testReport/junit/org.opensearch.rest.action.admin.cluster/RestNodesStatsActionTests/testIndexMetricsRequestWithoutIndicesMetric/

Expected behavior

Test should always pass

Additional Details

No response

[gaobinlong]

This flaky test relates to #13237 which introduced a new metric type caches to the node stats API, it will throw exception if the requested cache type is not valid:

OpenSearch/server/src/main/java/org/opensearch/rest/action/admin/cluster/RestNodesStatsAction.java

Line 191 in 14f1c43

throw new IllegalArgumentException(

, but when the caches metric is selected by the random method, it will enter this branch which handles caches metric type separately:

OpenSearch/server/src/main/java/org/opensearch/rest/action/admin/cluster/RestNodesStatsAction.java

Line 179 in 14f1c43

} else if (metrics.contains("caches")) {

, so at this point, java.lang.IllegalArgumentException: request [/_nodes/stats] contains unrecognized cache type: [store] will be thrown rather than request [/_nodes/stats] contains index metrics [store] but indices stats not requested.