[BUG] Flaky test ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting

[ashking94]

Describe the bug

Error Message
java.lang.AssertionError: MaxShardPerCluster 3 should be greater than or equal to MaxShardPerNode 1000
Stacktrace
java.lang.AssertionError: MaxShardPerCluster 3 should be greater than or equal to MaxShardPerNode 1000
	at __randomizedtesting.SeedInfo.seed([CF6070B40FF53C63:8586810AA2ABFADB]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.opensearch.cluster.shards.ClusterShardLimitIT.setMaxShardLimit(ClusterShardLimitIT.java:750)
	at org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting(ClusterShardLimitIT.java:259)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)

To Reproduce
Build failure - https://build.ci.opensearch.org/job/gradle-check/11004/testReport/junit/org.opensearch.cluster.shards/ClusterShardLimitIT/testCreateIndexWithMaxClusterShardSetting/

Auto cut - #6286

Expected behavior
The test should pass always.

[ashking94]

@RS146BIJAY Looks like this has come with #6143. Let's fix it or mute it.

[RS146BIJAY]

@ashking94 Thanks for reporting. I think this may have come from here. I think I am still using some random value in the test case. Will fix the test case.

[andrross]

Another failure: #8682 (comment)

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting" -Dtests.seed=99AA0399238084FC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sq -Dtests.timezone=Egypt -Druntime.java=20

org.opensearch.cluster.shards.ClusterShardLimitIT > testCreateIndexWithMaxClusterShardSetting FAILED
    java.lang.AssertionError: MaxShardPerCluster 2 should be greater than or equal to MaxShardPerNode 1000
        at __randomizedtesting.SeedInfo.seed([99AA0399238084FC:D34CF2278EDE4244]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.setMaxShardLimit(ClusterShardLimitIT.java:750)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting(ClusterShardLimitIT.java:259)