Allow TransportConfigUpdateAction when security config initialization has completed #3810

cwperks · 2023-12-08T16:01:23Z

Description

Introduces another variable on DynamicConfigFactory called bgThreadComplete that behaves differently than the initialized variable. bgThreadComplete is a flag that signals to TransportConfigUpdateAction that it can start accepting updates.

There are 2 ways the security index can be created from scratch:

If plugins.security.allow_default_init_securityindex is set to true it will create the security index and load all yaml files
If plugins.security.allow_default_init_securityindex is set to false, the security index is not created on bootstrap and requires a user to run securityadmin to initialize security. When securityadmin is utilized, the cluster does depend on TransportConfigUpdateAction to initialize security so there still needs to be an avenue where this can update the config before initialized is set to true

This PR sets bgThreadComplete to false on node startup and explicitly sets it to true once its ok for TransportConfigUpdateAction to start accepting transport actions. In case 2) up above, it can be set to true before DynamicConfigFactory is initialized so that it can accept requests from securityadmin.

Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Bug fix

Issues Resolved

Resolves [BUG] TransportConfigUpdateAction causing a race condition during node bootstrap leading to 403/500 errors #3204

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…tstrapped Signed-off-by: Craig Perkins <[email protected]>

Signed-off-by: Craig Perkins <[email protected]>

…ctions can pass Signed-off-by: Craig Perkins <[email protected]>

Signed-off-by: Craig Perkins <[email protected]>

src/main/java/org/opensearch/security/securityconf/DynamicConfigFactory.java

src/main/java/org/opensearch/security/action/configupdate/TransportConfigUpdateAction.java

src/integrationTest/java/org/opensearch/test/framework/cluster/LocalCluster.java

...nTest/java/org/opensearch/security/SecurityConfigurationBootstrapWithSecurityAdminTests.java

src/integrationTest/java/org/opensearch/security/SecurityConfigurationBootstrapTests.java

codecov · 2023-12-08T19:10:47Z

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (1f9edf4) 66.58% compared to head (4d53edc) 65.17%.
Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3810      +/-   ##
==========================================
- Coverage   66.58%   65.17%   -1.42%     
==========================================
  Files         298      298              
  Lines       21188    21218      +30     
  Branches     3453     3460       +7     
==========================================
- Hits        14109    13828     -281     
- Misses       5363     5674     +311     
  Partials     1716     1716

Files	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	`84.48% <100.00%> (+0.04%)`	⬆️
...ch/security/securityconf/DynamicConfigFactory.java	`55.62% <ø> (ø)`
...g/opensearch/security/support/ConfigConstants.java	`95.23% <ø> (ø)`
...tion/configupdate/TransportConfigUpdateAction.java	`95.45% <66.66%> (-4.55%)`	⬇️
...earch/security/privileges/PrivilegesEvaluator.java	`72.03% <0.00%> (-0.53%)`	⬇️
...ecurity/configuration/ConfigurationRepository.java	`69.11% <65.55%> (-10.35%)`	⬇️

... and 37 files with indirect coverage changes

Signed-off-by: Craig Perkins <[email protected]>

src/integrationTest/java/org/opensearch/security/ConfigurationFiles.java

src/integrationTest/java/org/opensearch/security/SecurityConfigurationBootstrapTests.java

Signed-off-by: Craig Perkins <[email protected]>

cwperks · 2023-12-19T18:09:46Z

Looking at the CI failures

into initialization-fix

Signed-off-by: Craig Perkins <[email protected]>

peternied

I'm concerned these test cases have will be flaky due to system performance differences on github runners. Lets make sure the validation is really solid since we are fixing a timing issue based problem.

…arer setup and tearDown Signed-off-by: Craig Perkins <[email protected]>

Signed-off-by: Peter Nied <[email protected]>

peternied

I've created a PR [1] that I think helps clean up the logic of this class so there is a set way that the background initialization is triggered - its always bugged me that we start a thread in the constructor but then 'restart it' after init is called. Let me know what you think of this approach.

[1] https:/cwperks/security/pull/17/files?diff=unified&w=1

src/main/java/org/opensearch/security/configuration/ConfigurationRepository.java

…eep security manager state Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Peter Nied <[email protected]>

…ternied/refactor-test' into HEAD Signed-off-by: Peter Nied <[email protected]>

peternied · 2024-01-04T21:27:54Z

@scrawfor99 Could you take another look?

stephen-crawford

Great work Craig!

opensearch-trigger-bot · 2024-01-05T20:17:14Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-2.x
# Create a new branch
git switch --create backport/backport-3810-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 045d4ef8a5d9bb8470e03c1b4d7cc68847b986cb
# Push it to GitHub
git push --set-upstream origin backport/backport-3810-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3810-to-2.x.

… initialization has completed (opensearch-project#3810) Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Peter Nied <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> (cherry picked from commit 045d4ef)

… initialization has completed (#3810) (#3927) ### Description - Backport of #3810 from 045d4ef ### Check List - [X] New functionality includes testing - [X] New functionality has been documented - [X] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https:/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Peter Nied <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Craig Perkins <[email protected]>

… has completed (opensearch-project#3810) Introduces another variable on DynamicConfigFactory called `bgThreadComplete` that behaves differently than the `initialized` variable. `bgThreadComplete` is a flag that signals to TransportConfigUpdateAction that it can start accepting updates. There are 2 ways the security index can be created from scratch: 1. If `plugins.security.allow_default_init_securityindex` is set to **true** it will create the security index and load all yaml files 2. If `plugins.security.allow_default_init_securityindex` is set to **false**, the security index is not created on bootstrap and requires a user to run securityadmin to initialize security. When securityadmin is utilized, the cluster does depend on [TransportConfigUpdateAction](https:/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/tools/SecurityAdmin.java#L975-L977) to initialize security so there still needs to be an avenue where this can update the config before `initialized` is set to **true** This PR sets `bgThreadComplete` to **false** on node startup and explicitly sets it to **true** once its ok for TransportConfigUpdateAction to start accepting transport actions. In case 2) up above, it can be set to **true** before DynamicConfigFactory is `initialized` so that it can accept requests from securityadmin. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix - Resolves opensearch-project#3204 - [X] New functionality includes testing - [ ] New functionality has been documented - [X] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https:/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Peter Nied <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> (cherry picked from commit 045d4ef)

… initialization has completed (#4115) Backport #3810 to 1.3 Testing performed with the configuration below: [1.3-latest.zip](https:/opensearch-project/security/files/14577043/1.3-latest.zip) This uses a configuration similar to the demo configuration, but with the following modifications that make a reproduction of this error more pronounced: ``` # plugins.security.allow_default_init_securityindex: true plugins.security.unsupported.restapi.allow_securityconfig_modification: true plugins.security.unsupported.load_static_resources: false ``` Steps to test: 1. Start the 3 node cluster 2. Exec into one of the nodes: ` docker exec -it 13-latest-opensearch-node2-1 /bin/bash` 3. Run securityadmin: `cd plugins/opensearch-security/tools && ./securityadmin.sh -cd ../securityconfig/ -icl -nhnv \ -cacert ../../../config/root-ca.pem \ -cert ../../../config/kirk.pem \ -key ../../../config/kirk-key.pem` 4. Exit the remote session: `exit` 5. Start infinite loop of config update in another terminal: `while true;do curl -ai -u "admin:admin" -k -X PATCH https://localhost:9200/_opendistro/_security/api/securityconfig -H 'Content-Type: application/json' -d'[{"op": "replace", "path": "/config/dynamic/authc/basic_internal_auth_domain/transport_enabled", "value": "true"}]'; done` 6. Restart one of the nodes Query the rebooted node and you will not get a response: ``` > curl -XGET https://admin:admin@localhost:9201 -k No response ``` If the cache is loaded correctly you will get a response, but access denied since static resource loading is disabled (this is expected and it means that internal users have been loaded into the cache): ``` > curl -XGET https://admin:admin@localhost:9200 -k {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403} ``` --------- Signed-off-by: Craig Perkins <[email protected]>

…config initialization… (#4116) Backport #3810 to opendistro-1.13 Testing performed with the configuration below: [opendistro-1.13-latest.zip](https:/opensearch-project/security/files/14578486/opendistro-1.13-latest.zip) This uses a configuration similar to the demo configuration, but with the following modifications that make a reproduction of this error more pronounced: ``` # opendistro_security.allow_default_init_securityindex: true opendistro_security.unsupported.restapi.allow_securityconfig_modification: true opendistro_security.unsupported.load_static_resources: false ``` Steps to test: 1. Start the 3 node cluster 2. Exec into one of the nodes: ` docker exec -it 13-latest-opensearch-node2-1 /bin/bash` 3. Run securityadmin: `cd plugins/opendistro_security/tools && chmod 777 * && ./securityadmin.sh -cd ../securityconfig/ -icl -nhnv \ -cacert ../../../config/root-ca.pem \ -cert ../../../config/kirk.pem \ -key ../../../config/kirk-key.pem` 4. Exit the remote session: `exit` 5. Start infinite loop of config update in another terminal: `while true;do curl -ai -u "admin:admin" -k -X PATCH https://localhost:9200/_opendistro/_security/api/securityconfig -H 'Content-Type: application/json' -d'[{"op": "replace", "path": "/config/dynamic/authc/basic_internal_auth_domain/transport_enabled", "value": "true"}]'; done` 6. Restart one of the nodes Query the rebooted node and you will not get a response: ``` > curl -XGET https://admin:admin@localhost:9201 -k Unauthorized ``` If the cache is loaded correctly you will get a response, but access denied since static resource loading is disabled (this is expected and it means that internal users have been loaded into the cache): ``` > curl -XGET https://admin:admin@localhost:9200 -k {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:monitor/main] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403} ``` ----- When checking out this branch you can use `mvn clean install -DskipTests` to build a jar. The jar is built in the `target` folder. Signed-off-by: Craig Perkins <[email protected]>

… has completed (opensearch-project#3810) Introduces another variable on DynamicConfigFactory called `bgThreadComplete` that behaves differently than the `initialized` variable. `bgThreadComplete` is a flag that signals to TransportConfigUpdateAction that it can start accepting updates. There are 2 ways the security index can be created from scratch: 1. If `plugins.security.allow_default_init_securityindex` is set to **true** it will create the security index and load all yaml files 2. If `plugins.security.allow_default_init_securityindex` is set to **false**, the security index is not created on bootstrap and requires a user to run securityadmin to initialize security. When securityadmin is utilized, the cluster does depend on [TransportConfigUpdateAction](https:/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/tools/SecurityAdmin.java#L975-L977) to initialize security so there still needs to be an avenue where this can update the config before `initialized` is set to **true** This PR sets `bgThreadComplete` to **false** on node startup and explicitly sets it to **true** once its ok for TransportConfigUpdateAction to start accepting transport actions. In case 2) up above, it can be set to **true** before DynamicConfigFactory is `initialized` so that it can accept requests from securityadmin. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Bug fix - Resolves opensearch-project#3204 - [X] New functionality includes testing - [ ] New functionality has been documented - [X] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https:/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: Peter Nied <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]>

cwperks added 3 commits December 6, 2023 16:37

Only allow TransportConfigUpdateAction after node has been set as boo…

7ba39c9

…tstrapped Signed-off-by: Craig Perkins <[email protected]>

Set in finally block and change name to bgThreadComplete

d05da1a

Signed-off-by: Craig Perkins <[email protected]>

Add tests that ensure the bg thread completes during bootstrap

58d6f45

Signed-off-by: Craig Perkins <[email protected]>

cwperks requested review from cliu123, DarshitChanpura, davidlago, peternied, RyanL1997, stephen-crawford, reta and willyborankin as code owners December 8, 2023 16:01

cwperks added 3 commits December 8, 2023 11:28

Add test to provide initialize with securityadmin works

c2423b7

Signed-off-by: Craig Perkins <[email protected]>

Set background init to false to ensure that transport config update a…

d86feed

…ctions can pass Signed-off-by: Craig Perkins <[email protected]>

Update SecurityConfigurationTests

9606f38

Signed-off-by: Craig Perkins <[email protected]>

peternied reviewed Dec 8, 2023

View reviewed changes

cwperks added 7 commits December 11, 2023 12:59

Use CompletableFuture

0b764f1

Signed-off-by: Craig Perkins <[email protected]>

Wrap in doPrivileged

83961f0

Signed-off-by: Craig Perkins <[email protected]>

doPrivileged

b211302

Signed-off-by: Craig Perkins <[email protected]>

Wrap with doPrivileged

0613161

Signed-off-by: Craig Perkins <[email protected]>

Assert not initialized

258fb28

Signed-off-by: Craig Perkins <[email protected]>

Rename beforeClass method

91cf472

Signed-off-by: Craig Perkins <[email protected]>

Add AccessControllerWrappedThread

06b7513

Signed-off-by: Craig Perkins <[email protected]>

stephen-crawford reviewed Dec 11, 2023

View reviewed changes

cwperks added 2 commits December 12, 2023 15:27

Use threadContext to carry context

a3398e0

Signed-off-by: Craig Perkins <[email protected]>

Merge branch 'main' into initialization-fix

07902e2

Signed-off-by: Craig Perkins <[email protected]>

cwperks added 3 commits December 19, 2023 13:10

Merge branch 'main' into initialization-fix

d80e98d

Merge branch 'initialization-fix' of https:/cwperks/security

67272e0

into initialization-fix

Small update

7b73715

Signed-off-by: Craig Perkins <[email protected]>

Remote restart

4e32d0f

Signed-off-by: Craig Perkins <[email protected]>

cwperks force-pushed the initialization-fix branch from 5a2ff86 to 4e32d0f Compare December 20, 2023 18:54

peternied requested changes Dec 20, 2023

View reviewed changes

cwperks and others added 3 commits December 22, 2023 13:23

Merge branch 'main' into initialization-fix

b74d7f4

Add setting to delay initialization in order to write a test with cle…

b3b6de7

…arer setup and tearDown Signed-off-by: Craig Perkins <[email protected]>

Only start initalization the configuration once

9442ad4

Signed-off-by: Peter Nied <[email protected]>

peternied reviewed Jan 4, 2024

View reviewed changes

peternied and others added 4 commits January 3, 2024 20:11

Switch back to thread for execution of the cluster configuration to k…

3fe1ddb

…eep security manager state Signed-off-by: Peter Nied <[email protected]>

Fix spotless spacing

66dcfce

Signed-off-by: Peter Nied <[email protected]>

Refactor bootstrap tests to share setup steps and add prevalidation

2f552d3

Signed-off-by: Peter Nied <[email protected]>

Merge remote-tracking branches 'peternied/initialization-fix' and 'pe…

4d53edc

…ternied/refactor-test' into HEAD Signed-off-by: Peter Nied <[email protected]>

peternied approved these changes Jan 4, 2024

View reviewed changes

peternied added the backport 2.x backport to 2.x branch label Jan 4, 2024

peternied changed the title ~~Only allow TransportConfigUpdateAction after bgThread for initialization has completed~~ Allow TransportConfigUpdateAction when security config initialization has completed Jan 5, 2024

stephen-crawford approved these changes Jan 5, 2024

View reviewed changes

stephen-crawford merged commit 045d4ef into opensearch-project:main Jan 5, 2024
81 of 83 checks passed

opensearch-trigger-bot bot added the backport-failed label Jan 5, 2024

peternied mentioned this pull request Jan 5, 2024

[Backport 2.x] Allow TransportConfigUpdateAction when security config initialization has completed (#3810) #3927

Merged

3 tasks

This was referenced Mar 12, 2024

[Backport 1.3] Allow TransportConfigUpdateAction when security config initialization has completed #4115

Merged

[Backport ODFE-1.13] Allow TransportConfigUpdateAction when security config initialization… #4116

Merged

cwperks mentioned this pull request Mar 19, 2024

[Backport 1.3] Integration Tests for Security Config Initialization #4134

Merged

3 tasks

DarshitChanpura removed the backport-failed label Jul 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow TransportConfigUpdateAction when security config initialization has completed #3810

Allow TransportConfigUpdateAction when security config initialization has completed #3810

cwperks commented Dec 8, 2023 •

edited by peternied

Loading

codecov bot commented Dec 8, 2023 •

edited

Loading

cwperks commented Dec 19, 2023

peternied left a comment

peternied left a comment

peternied commented Jan 4, 2024

stephen-crawford left a comment

opensearch-trigger-bot bot commented Jan 5, 2024

Allow TransportConfigUpdateAction when security config initialization has completed #3810

Allow TransportConfigUpdateAction when security config initialization has completed #3810

Conversation

cwperks commented Dec 8, 2023 • edited by peternied Loading

Description

Issues Resolved

Check List

codecov bot commented Dec 8, 2023 • edited Loading

Codecov Report

cwperks commented Dec 19, 2023

peternied left a comment

Choose a reason for hiding this comment

peternied left a comment

Choose a reason for hiding this comment

peternied commented Jan 4, 2024

stephen-crawford left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Jan 5, 2024

cwperks commented Dec 8, 2023 •

edited by peternied

Loading

codecov bot commented Dec 8, 2023 •

edited

Loading