[BUG] repository-gcs java.lang.SecurityException

[mickaelstrg]

Describe the bug
Performing REST calls in an existing snapshot gcs repository throws 500 errors with java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")

To Reproduce
Steps to reproduce the behavior:

Unclear, everything snapshot-related used to work fine till it did not anymore.

The repository-gcs plugin should be already correctly installed. Snapshots are stored in a GCS bucket for which authentication is done with workload identity (once again, this used to work fine and nothing was changed about that configuration).

1. Create a GCS snapshot repository.
2. Have some snapshots created in the repository (when it was still working)
3. Try to e.g. list snapshots in the repo: curl -sk -u admin:admin "https://localhost:9200/_cat/snapshots/repo_name"

{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[repo_name] Unexpected exception when loading repository data"
      }
    ],
    "type": "repository_exception",
    "reason": "[repo_name] Unexpected exception when loading repository data",
    "caused_by": {
      "type": "security_exception",
      "reason": "access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")"
    }
  },
  "status": 500
}

Stacktrace:

opensearch-cluster-master-1 opensearch [2023-10-09T12:43:16,136][WARN ][r.suppressed             ] [opensearch-cluster-master-1] path: /_snapshot/repo_name/_all, params: {repository=repo_name, snapshot=_all}
opensearch-cluster-master-1 opensearch org.opensearch.repositories.RepositoryException: [agents] Unexpected exception when loading repository data
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1739) ~[opensearch-2.8.0.jar:2.8.0]
opensearch-cluster-master-1 opensearch     at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) [opensearch-2.8.0.jar:2.8.0]
opensearch-cluster-master-1 opensearch     at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.8.0.jar:2.8.0]
opensearch-cluster-master-1 opensearch     at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.8.0.jar:2.8.0]
opensearch-cluster-master-1 opensearch     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
opensearch-cluster-master-1 opensearch     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
opensearch-cluster-master-1 opensearch     at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-cluster-master-1 opensearch Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
opensearch-cluster-master-1 opensearch     at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
opensearch-cluster-master-1 opensearch     at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
opensearch-cluster-master-1 opensearch     at java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
opensearch-cluster-master-1 opensearch     at java.lang.Class.checkMemberAccess(Class.java:3051) ~[?:?]
opensearch-cluster-master-1 opensearch     at java.lang.Class.getDeclaredFields(Class.java:2369) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:175) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.ClassInfo.of(ClassInfo.java:90) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:198) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.ClassInfo.of(ClassInfo.java:90) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.ClassInfo.<init>(ClassInfo.java:198) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.ClassInfo.of(ClassInfo.java:90) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.GenericData.<init>(GenericData.java:74) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.util.GenericData.<init>(GenericData.java:55) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:195) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:176) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:125) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.api.client.http.GenericUrl.<init>(GenericUrl.java:108) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:493) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31) ~[?:?]
opensearch-cluster-master-1 opensearch     at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78) ~[?:?]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:232) ~[?:?]
Stream closed EOF for opensearch/opensearch-cluster-master-1 (init-sysctl)
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:171) ~[?:?]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:128) ~[?:?]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:133) ~[?:?]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:222) ~[?:?]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:85) ~[?:?]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:1890) ~[opensearch-2.8.0.jar:2.8.0]
opensearch-cluster-master-1 opensearch     at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1701) ~[opensearch-2.8.0.jar:2.8.0]
opensearch-cluster-master-1 opensearch     ... 6 more

Expected behavior
Listing snapshots in a snapshot repository should be possible without errors.

Plugins

• repository-gcs
• prometheus-exporter 2.8.0

Screenshots
N/A

Host/Environment (please complete the following information):

• OS: official opensearch docker image running in a GKE cluster
• Version: 2.8.0

Additional context
Everything was working till I set up opensearch for TLS with custom CA & cert used on all nodes. Could also be a coincidence, I don't know. API calls with https work fine for other parts of the software but not for snapshots anymore.

I've looked for quite some time for other issues similar to this one but couldn't find anything that worked. I tried restarting the opensearch cluster to no avail, tried to reload secure settings to no avail, I'm out of ideas :(

[dblock]

Looks like this is similar to #4269? It was fixed then. I'd double check that this is still the case on 2.10 first, then we can try to debug the same way as in that issue.

[mickaelstrg]

Thanks for the reply @dblock.

I updated the cluster to 2.10.0 but the same issue was still there. I then tried to, just like in #4269, update the security.policy file with:

grant {
    permission java.lang.RuntimePermission "accessDeclaredMembers";
}

in the opensearch-2.10.0.jar file and repack it and this time, I got a new error message slightly different from the previous one:

❯ curl -sk -u admin:admin "https://localhost:9200/_snapshot/repo_name/_all" | jq .
{
  "error": {
    "root_cause": [
      {
        "type": "repository_exception",
        "reason": "[repo_name] Unexpected exception when loading repository data"
      }
    ],
    "type": "repository_exception",
    "reason": "[repo_name] Unexpected exception when loading repository data",
    "caused_by": {
      "type": "security_exception",
      "reason": "access denied (\"java.lang.reflect.ReflectPermission\" \"suppressAccessChecks\")"
    }
  },
  "status": 500
}

I then tried to add permission java.lang.reflect.ReflectPermission "suppressAccessChecks"; to the same grant block in the security.policy file, repacked the jar, injected it but the exact same error (the one with suppressAccessChecks) was showing up.

[dblock]

🤔 I don't know - @reta you have some more experience with these maybe you do?

[reta]

🤔 I don't know - @reta you have some more experience with these maybe you do?

It seems like we need to wrap it up in the privileged call, I will take care of it