Add integration tests for remote store restore flow

[BhumikaSaini-Amazon]

Description

Add the following test scenarios to the suite:

1. Restoring an index with >= 1 replica - 1 for each of the restore source / data loss (if any) combinations (RSS only, RSS+RTS both, flushed data, refreshed data, unrefreshed data)
2. Specifying indices using wildcards.
3. Excluding indices using wildcards.
4. Restoring multiple indices via single request.
5. Restoring only few of the red remote-enabled indices.

Related Issues

#8483

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19252/
• CommitID: 8162027
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19253/
• CommitID: 63ce4c7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19298/
• CommitID: 08efbd9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19299/
• CommitID: 0361ada
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[BhumikaSaini-Amazon]

testRTSRestoreWithCommittedDataNotAllRedRemoteIndices is failing even though testRemoteTranslogRestoreWithCommittedData passes. Need to check why.

Resolved, indexSingleDoc() was always using INDEX_NAME

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20559/
• CommitID: 6f3b206
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20564/
• CommitID: a0e0e13
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20581/
• CommitID: 74931f6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

@BhumikaSaini-Amazon Based on the analysis that I did, this is what I have found. This issue would not happen for indexes with just 1 shard as the node that holds the shard is stopped and there is no history of retention leases (of the shard that was residing on the stopped node) after we close index and start the remote store restore.

For the indexes that have more than 1 shard, this issue happens when the syncRetentionLeases async task (that runs periodically) persists the retention leases to the local disk.

OpenSearch/server/src/main/java/org/opensearch/index/seqno/RetentionLeaseSyncAction.java

Lines 184 to 196 in 83a4b6e

	protected void dispatchedShardOperationOnPrimary(
	Request request,
	IndexShard primary,
	ActionListener<PrimaryResult<Request, Response>> listener
	) {
	ActionListener.completeWith(listener, () -> {
	assert request.waitForActiveShards().equals(ActiveShardCount.NONE) : request.waitForActiveShards();
	Objects.requireNonNull(request);
	Objects.requireNonNull(primary);
	primary.persistRetentionLeases();
	return new WritePrimaryResult<>(request, new Response(), null, null, primary, getLogger());
	});
	}

Afterwards, on cluster state update for restoring the remote index, IndexShard is created and then recovery starts on the newly created IndexShard object.

OpenSearch/server/src/main/java/org/opensearch/indices/IndicesService.java

Lines 930 to 967 in 83a4b6e

	public IndexShard createShard(
	final ShardRouting shardRouting,
	final SegmentReplicationCheckpointPublisher checkpointPublisher,
	final PeerRecoveryTargetService recoveryTargetService,
	final RecoveryListener recoveryListener,
	final RepositoriesService repositoriesService,
	final Consumer<IndexShard.ShardFailure> onShardFailure,
	final Consumer<ShardId> globalCheckpointSyncer,
	final RetentionLeaseSyncer retentionLeaseSyncer,
	final DiscoveryNode targetNode,
	final DiscoveryNode sourceNode,
	final RemoteRefreshSegmentPressureService remoteRefreshSegmentPressureService
	) throws IOException {
	Objects.requireNonNull(retentionLeaseSyncer);
	ensureChangesAllowed();
	IndexService indexService = indexService(shardRouting.index());
	assert indexService != null;
	RecoveryState recoveryState = indexService.createRecoveryState(shardRouting, targetNode, sourceNode);
	IndexShard indexShard = indexService.createShard(
	shardRouting,
	globalCheckpointSyncer,
	retentionLeaseSyncer,
	checkpointPublisher,
	remoteRefreshSegmentPressureService
	);
	indexShard.addShardFailureCallback(onShardFailure);
	indexShard.startRecovery(recoveryState, recoveryTargetService, recoveryListener, repositoriesService, mapping -> {
	assert recoveryState.getRecoverySource().getType() == RecoverySource.Type.LOCAL_SHARDS
	: "mapping update consumer only required by local shards recovery";
	client.admin()
	.indices()
	.preparePutMapping()
	.setConcreteIndex(shardRouting.index()) // concrete index - no name clash, it uses uuid
	.setSource(mapping.source().string(), XContentType.JSON)
	.get();
	}, this);
	return indexShard;
	}

During the recovery from remote store, the innerOpenEngineAndTranslog method of IndexShard is invoked where the retentionLeases are restored from the local disk back to in-memory replication tracker.

OpenSearch/server/src/main/java/org/opensearch/index/shard/IndexShard.java

Lines 2310 to 2325 in 83a4b6e

	private void innerOpenEngineAndTranslog(LongSupplier globalCheckpointSupplier, boolean syncFromRemote) throws IOException {
	assert Thread.holdsLock(mutex) == false : "opening engine under mutex";
	if (state != IndexShardState.RECOVERING) {
	throw new IndexShardNotRecoveringException(shardId, state);
	}
	final EngineConfig config = newEngineConfig(globalCheckpointSupplier);
	// we disable deletes since we allow for operations to be executed against the shard while recovering
	// but we need to make sure we don't loose deletes until we are done recovering
	config.setEnableGcDeletes(false);
	updateRetentionLeasesOnReplica(loadRetentionLeases());
	assert recoveryState.getRecoverySource().expectEmptyRetentionLeases() == false || getRetentionLeases().leases().isEmpty()
	: "expected empty set of retention leases with recovery source ["
	+ recoveryState.getRecoverySource()
	+ "] but got "
	+ getRetentionLeases();

This issue is flaky due to the nature of async sync retention leases. As for now, I am creating an issue to remove retention leases when index has remote translog enabled. Meanwhile, we can safely override the expectEmptyRetentionLeases of RemoteStoreRecoverySource class.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20587/
• CommitID: 990b160
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/20597/
• CommitID: dd1968d

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8484-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 27a14c7c811f930b2b36c13f20e1371c6206dd51
# Push it to GitHub
git push --set-upstream origin backport/backport-8484-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8484-to-2.x.