[BUG] org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue flaky

[andrross]

See error here: https://build.ci.opensearch.org/job/gradle-check/6130/consoleFull

Not reproducible.

FYI @xuezhou25 This looks like it might be a test you recently introduced?

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue" -Dtests.seed=C7E67525143CA276 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=bg-BG -Dtests.timezone=Pacific/Marquesas -Druntime.java=17

org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT > testIndexDeletionDuringSnapshotCreationInQueue FAILED
    java.lang.IllegalStateException: Cannot cleanup [test-repo] - a snapshot is currently running in [SnapshotsInProgress[test-snap-2]]
        at __randomizedtesting.SeedInfo.seed([C7E67525143CA276:FFCD0E8BC360C7E7]:0)
        at org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$2.execute(TransportCleanupRepositoryAction.java:254)
        at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
        at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:843)
        at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:414)
        at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:285)
        at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:196)
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:176)
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:214)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
        at java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.****/java.lang.Thread.run(Thread.java:833)

[dblock]

#5141

[dreamer-89]

Curious on this test failure as it is the second top contributor to gradle check failures in last two months.

[mch2]

#6015

[gbbafna]

#6257

[ashking94]

#10819 (comment) shows as failed in summary, but the test has actually passed.

[shwetathareja]

Latest failure - #10685 (comment)

java.lang.IllegalStateException: Cannot cleanup [test-repo] - a snapshot is currently running in [SnapshotsInProgress[test-snap-2]]
	at __randomizedtesting.SeedInfo.seed([A2454F994289A0AE:9A6E343795D5C53F]:0)
	at org.opensearch.action.admin.cluster.repositories.cleanup.TransportCleanupRepositoryAction$2.execute(TransportCleanupRepositoryAction.java:244)
	at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65)
	at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874)
	at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424)
	at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295)
	at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206)
	at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204)
	at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

[ashking94]

#10939 (comment)

[dreamer-89]

Ran this test locally with/without failing test seed but without any success. I see this test though is still failing occasionally where I ran the flaky finder script from starting build number 29000 (Oct 25)

➜  workspace ruby ~/Desktop/Older\ Desktop/flaky-finder.rb --s 29000 --e 31322                                
Will crawl builds from 29000 to 31322
------------------
124 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing (30490,30501,30501,30518,30520,30523,30535,30536,30536,30567,30568,30573,30581,30583,30585,30593,30599,30607,30607,30617,30625,30630,30630,30630,30631,30640,30641,30650,30655,30661,30674,30682,30682,30684,30691,30693,30697,30714,30735,30740,30741,30743,30758,30761,30786,30796,30813,30814,30823,30837,30839,30881,30890,30896,30896,30897,30898,30921,30922,30922,30931,30932,30938,30939,30942,30945,30945,30949,30951,30959,30963,30964,30968,30976,30981,30981,31005,31006,31008,31010,31025,31047,31058,31068,31077,31084,31089,31092,31105,31109,31116,31119,31121,31128,31130,31134,31153,31165,31168,31168,31168,31189,31190,31192,31215,31215,31217,31220,31229,31231,31232,31235,31235,31252,31263,31280,31280,31284,31295,31304,31305,31306,31320,31322)
77 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all} (29035,29108,29112,29250,29251,29266,29266,29336,29408,29554,29857,29876,29893,29900,29942,29997,29997,30019,30026,30040,30047,30060,30070,30112,30185,30188,30200,30243,30245,30331,30347,30348,30444,30469,30502,30502,30507,30507,30508,30599,30601,30638,30639,30660,30693,30720,30747,30758,30766,30770,30881,30936,30939,31008,31032,31032,31053,31082,31088,31089,31089,31102,31107,31107,31107,31114,31117,31217,31229,31229,31242,31252,31255,31277,31289,31312,31315)
69 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled (29186,29214,29375,29404,29406,29458,29476,29574,29579,29587,29688,29773,29797,29799,29801,29802,29803,29816,29870,29906,29931,29933,29936,29940,29942,30009,30030,30072,30099,30113,30221,30257,30264,30268,30283,30294,30421,30423,30432,30438,30500,30560,30659,30689,30724,30751,30796,30807,30820,30852,30871,30895,30898,30968,30981,30998,31006,31014,31016,31049,31078,31093,31104,31107,31108,31179,31191,31232,31320)
29 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled (29107,29327,29498,29498,29617,29617,29772,29873,29905,29913,29939,29943,29956,29956,30030,30113,30143,30200,30209,30300,30491,30628,30637,30940,31124,31245,31272,31279,31292)
26 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards (29074,29174,29204,29406,29529,29801,29893,29921,30006,30019,30147,30182,30343,30348,30381,30393,30433,30490,30509,30540,30974,31080,31090,31232,31284,31302)
24 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod (29707,29707,29753,29753,29842,29842,30119,30119,30149,30149,30390,30390,30401,30401,30720,30720,30720,30720,30826,30826,31114,31114,31241,31241)
19 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness (29037,29083,29171,29291,29496,29719,29747,29779,30165,30220,30286,30292,30421,30427,30768,30777,30805,30868,30932)
18 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs (29200,29209,29296,29336,29384,29608,29779,29935,30098,30235,30336,30375,30427,30431,30444,30580,30850,30932)
18 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards (29038,29109,29119,29162,29217,29305,29327,29447,29528,29707,29867,30057,30188,30381,30555,31085,31091,31140)
18 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimarySingleReplica (29129,29235,29380,29419,29443,29663,30089,30132,30193,30225,30275,30279,30286,30480,30518,30610,30654,31178)
18 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob (29115,29119,29227,29251,29251,29254,29489,29529,29558,29617,29698,29850,30213,30448,30453,30481,30562,30935)
16 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation (29140,29571,29783,29943,30010,30194,30324,30381,30674,30743,30910,30938,31089,31104,31194,31252)
12 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown (29707,29753,29842,30119,30149,30390,30401,30720,30720,30826,31114,31241)
12 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose (29219,29492,29661,29907,30147,30185,30292,30544,30608,30761,30768,30848)
12 org.opensearch.index.shard.RemoteIndexShardTests.classMethod (29017,29017,29048,29048,29496,29496,29596,29596,29596,30908,30908,30908)
11 org.opensearch.cluster.routing.MovePrimaryFirstTests.testClusterGreenAfterPartialRelocation (29053,29072,29110,29126,29145,29146,29182,29193,29211,29228,29229)
11 org.opensearch.cluster.routing.MovePrimaryFirstTests.classMethod (29053,29072,29110,29126,29145,29146,29182,29193,29211,29228,29229)
10 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled (29186,29294,29413,29463,29528,29528,30205,30404,30937,31190)
10 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod (30659,30659,30895,30895,31003,31003,31109,31109,31114,31114)
9 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex (29066,29112,29247,29440,30133,30352,30534,30781,30891)
9 org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled (29264,29592,29779,30094,30238,30247,30336,30616,31079)
8 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries (29140,29329,29584,29802,29932,30015,31006,31304)
6 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testTrackerData (29815,30000,30011,30101,30820,30892)
6 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled (29046,29051,29119,29212,29272,29392)
6 org.opensearch.client.PitIT.testDeleteAllAndListAllPits (29171,29245,29327,29408,29408,29584)
6 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount (29724,30217,30344,30606,31255,31278)
5 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue (29672,29759,30422,30452,30632)
5 org.opensearch.indices.replication.SegmentReplicationIT.classMethod (29102,29130,30009,30347,30910)
5 org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew (29489,29574,30109,30353,30969)
5 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=pit/10_basic/Delete all} (29434,30560,30599,30892,31192)
4 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadRangeBlobWithRetries (29341,30073,30407,31293)
4 org.opensearch.test.rest.ClientYamlTestSuiteIT.classMethod (29664,29753,30121,30223)
4 org.opensearch.index.shard.RemoteIndexShardTests.testSegRepSucceedsOnPreviousCopiedFiles (29017,29048,29078,29496)
4 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testOverriddenBufferInterval (29483,29858,30027,30357)
4 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshAfterCommit (30895,31003,31109,31114)
4 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseAllShards (29049,30070,30301,30761)
4 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals} (30223,30754,30945,31059)
4 org.opensearch.action.admin.indices.create.CreateIndexIT.testCreateAndDeleteIndexConcurrently (29337,30193,30638,31079)
4 org.opensearch.action.admin.indices.create.CreateIndexIT.classMethod (29337,30193,30638,31079)
3 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testWriteRead (30411,30643,30835)
3 org.opensearch.index.shard.RemoteIndexShardTests.testNoFailuresOnFileReads (29046,29089,29089)
3 org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureLastSuccessfulSettingsUpdate (29207,29851,30042)
3 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessAfterFailureInFirstAttemptAfterSnapshotAndMetadataUpload (29245,30452,30659)
3 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRestartPrimary_NoReplicas (29543,29594,30340)
3 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringQueryPhase (29545,29594,29940)
3 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testDefaultBufferInterval (29782,29857,31104)
3 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadNonexistentBlobThrowsNoSuchFileException (29840,30116,31168)
3 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation (29963,30647,31215)
3 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.classMethod (30101,31217,31217)
3 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase (30217,30445,30590)
3 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals} (30468,30898,31165)
3 org.opensearch.index.reindex.UpdateByQueryBasicTests.testMultipleSources (30547,31284,31284)
3 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadBlobWithRetries (30592,30935,31264)
2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnSecondAttempt (29523,30743)
2 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testRelocateWhileContinuouslyIndexingAndWaitingForRefresh (30438,31146)
2 org.opensearch.indices.state.CloseWhileRelocatingShardsIT.testCloseWhileRelocatingShards (29910,29950)
2 org.opensearch.index.ShardIndexingPressureIT.testShardIndexingPressureTrackingDuringBulkWrites (29915,30998)
2 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.classMethod (29753,30223)
2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnFirstAttempt (30570,30601)
2 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback (29950,30246)
2 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testRequestDurabilityWhenRestrictSettingExplicitFalse (29282,29940)
2 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.classMethod (29753,30223)
2 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityThrowsExceptionWhenRestrictSettingTrue (29483,29664)
2 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase (29574,29944)
2 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search.aggregation/20_terms/string profiler via global ordinals} (29574,30059)
2 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testRequestDurabilityWhenRestrictSettingImplicitFalse (29592,31145)
2 org.opensearch.remotestore.CreateRemoteIndexIT.testDefaultRemoteStoreNoUserOverride (29037,31280)
2 org.opensearch.client.PitIT.classMethod (29408,29408)
2 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"false"}} (29965,30285)
2 org.opensearch.index.reindex.DeleteByQueryBasicTests.testDeleteByQueryWithMultipleIndices (29161,30079)
1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=indices.get_field_mapping/10_basic/Get field mapping with include_defaults} (30121)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication (31217)
1 org.opensearch.aliases.IndexAliasesIT.testSameAlias (29164)
1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=indices.rollover/40_mapping/Typeless mapping} (30223)
1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=indices.get/10_basic/Should return test_index_2 if expand_wildcards=open} (30223)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testList (31289)
1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromLocalNode (29108)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles (31305)
1 org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob (31308)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testRealtimeMultiGetRequestsUnsuccessful (30608)
1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica (30617)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testDeleteBlobs (30661)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats (30704)
1 org.opensearch.index.ShardIndexingPressureSettingsIT.testShardIndexingPressureEnforcedEnabledDisabledSetting (30710)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testScrollCreatedOnReplica (30720)
1 org.opensearch.index.reindex.DeleteByQueryBasicTests.testDeleteByQueryOnReadOnlyAllowDeleteIndex (29023)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles (30897)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testEnforcedCooldownPeriod (30897)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testReadNonExistingPath (30897)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testList (30897)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testDeleteBlobs (30897)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.testSnapshotAndRestore (30897)
1 org.opensearch.index.shard.RemoteIndexShardTests.testSegmentReplication_With_EngineClosedConcurrently (29596)
1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterCommit (29608)
1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=indices.delete/10_basic/Delete index against alias -  multiple indices} (29664)
1 org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testShardAlreadyReplicating (29554)
1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityWhenRestrictSettingImplicitFalse (29442)
1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/260_sort_mixed/search across indices with mixed long and double and float numeric types} (29384)
1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.get_mapping/10_basic/Get /_mapping} (29753)
1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=search/160_exists_query/Test exists query on mapped byte field} (29753)
1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search/160_exists_query/Test exists query on mapped byte field} (29753)
1 org.opensearch.repositories.s3.S3BlobStoreRepositoryTests.classMethod (30897)
1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityWhenRestrictSettingExplicitFalse (29791)
1 org.opensearch.index.shard.RemoteIndexShardTests.testReplication (30908)
1 org.opensearch.indices.replication.RemoteStoreReplicationSourceTests.testGetSegmentFilesAlreadyExists (29906)
1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsResponseFromAllNodes (29943)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.classMethod (29950)
1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase (29955)
1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttempt (31025)
1 org.opensearch.snapshots.CloneSnapshotIT.testClonePreventsSnapshotDelete (30047)
1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaAlreadyAtCheckpoint (30101)
1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore (31084)