Fix flaky test SlowIntegrationTests.testDelayInSecurityIndexInitialization

[cwperks]

Description

Explicitly adds a command after setup to create an index and wait for cluster green which produces the expected IOException.

The pseudo code of this test is:

1. Setup a cluster, but forbid allocating any shards so cluster is red
2. If cluster is red as expected, then remove the restriction on shard allocation and wait for cluster to turn green
3. Wait another 10s to ensure that the security index is initialized.

I believe this test is not succeeding, because the cluster is coming up as green initially since there is no data (and hence no shards) in the cluster. By adding an explicit command to create an index, the test ensures that the shards are not being allocated and cluster is indeed red.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Test Fix

Issues Resolved

• Resolves [BUG] Intermittent test failure SlowIntegrationTests.testDelayInSecurityIndexInitialization #2230

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #3763 (34da97b) into main (4234823) will decrease coverage by 0.03%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3763      +/-   ##
==========================================
- Coverage   65.27%   65.25%   -0.03%     
==========================================
  Files         298      298              
  Lines       21159    21159              
  Branches     3455     3455              
==========================================
- Hits        13812    13807       -5     
- Misses       5653     5656       +3     
- Partials     1694     1696       +2     

see 3 files with indirect coverage changes

[DarshitChanpura]

ty @cwperks! This will resolve a big CI blocker.
Can we confirm the fix by running across multiple CI runs?

[cwperks]

FYI @DarshitChanpura this passed the 10x check. I will remove the line to repeat the test.

[cwperks]

@peternied @DarshitChanpura This PR adds back in the SlowIntegrationTests.testDelayInSecurityIndexInitialization that was removed in #3706, but updates the test to make it stable. IMO I would like to keep the test in, but have it been a reliable test.

[cwperks]

@peternied Can I get another review of this? All comments have been addressed. This will re-instate a (flaky) test that was recently removed, but stabilizes the test.

[stephen-crawford]

Thanks Craig

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-2.x
# Create a new branch
git switch --create backport/backport-3763-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7498eb00679c307b75db80f54476a01ace48fae9
# Push it to GitHub
git push --set-upstream origin backport/backport-3763-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3763-to-2.x.

[cwperks]

I will create a manual backport