[BUG] Getting security exception due to access denied "java" #5331

max-frank · 2022-11-22T03:21:26Z

I am experiencing similar issues with GCS repositories as were discussed in #4269 for S3 repositories.

In 2.2.x the GCS snapshots were working fine running with Workload Identity in a GKE cluster, but starting with 2.3.0 i started seeing the same java security policy issues. In 2.3.0 I was still able to take snapshots even though it showed that all indices failed. With 2.4.0 I cannot even load the list of repositories anymore.

To Reproduce
Steps to reproduce the behavior:

Setup GKE with workload identity and map KSA to GSA with bucket write/read/etc.
Deploy Opensearch with KSA and repository-gcs installed
Try to snapshot/load repositories/etc.
Errors

Note that all the missing permissions being complained about are configured in the plugins security policy file already by default.

trying to load repositories

[2022-11-22T03:14:17,406][WARN ][r.suppressed             ] [opensearch-cluster-main-0] path: /_snapshot/opencti-opensearch-backup-repository/_all, params: {ignore_unavailable=true, repository=opencti-opensearch-backup-repository, snapshot=_all}
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-managers-2][192.168.34.14:9300][cluster:admin/snapshot/get]
Caused by: org.opensearch.repositories.RepositoryException: [opencti-opensearch-backup-repository] Unexpected exception when loading repository data
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1734) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.SecurityException: access denied ("java.net.SocketPermission" "metadata.google.internal:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:919) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:619) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:279) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:384) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:406) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31) ~[?:?]
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:232) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:171) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:128) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:133) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:222) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:85) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:1885) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1696) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

trying to get info on specific snapshot

[2022-11-22T03:17:58,284][WARN ][r.suppressed             ] [opensearch-cluster-main-0] path: /_snapshot/opencti-opensearch-backup-repository/all, params: {pretty=true, repository=opencti-opensearch-backup-repository, snapshot=all}
org.opensearch.transport.RemoteTransportException: [opensearch-cluster-managers-2][192.168.34.14:9300][cluster:admin/snapshot/get]
Caused by: org.opensearch.repositories.RepositoryException: [opencti-opensearch-backup-repository] Unexpected exception when loading repository data
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1734) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.SecurityException: access denied ("java.net.SocketPermission" "metadata.google.internal:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
        at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:416) ~[?:?]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:919) ~[?:?]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:619) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:279) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:384) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:406) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1309) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1057) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) ~[?:?]
        at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[?:?]
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectIdFromMetadataServer(ServiceOptions.java:503) ~[?:?]
        at com.google.cloud.ServiceOptions.getAppEngineProjectId(ServiceOptions.java:472) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProjectId(ServiceOptions.java:379) ~[?:?]
        at com.google.cloud.ServiceOptions.getDefaultProject(ServiceOptions.java:356) ~[?:?]
        at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:302) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:83) ~[?:?]
        at com.google.cloud.storage.StorageOptions.<init>(StorageOptions.java:31) ~[?:?]
        at com.google.cloud.storage.StorageOptions$Builder.build(StorageOptions.java:78) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createStorageOptions(GoogleCloudStorageService.java:232) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.createClient(GoogleCloudStorageService.java:171) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageService.client(GoogleCloudStorageService.java:128) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.client(GoogleCloudStorageBlobStore.java:133) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobStore.readBlob(GoogleCloudStorageBlobStore.java:222) ~[?:?]
        at org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:85) ~[?:?]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:1885) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1696) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.4.0.jar:2.4.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.4.0.jar:2.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Expected behavior
Able to take, restore, etc. snapshots in combination with a GCS repository while running with Workload Identity (or similar).

Plugins

repository-gcs

Screenshots

Host/Environment (please complete the following information):

OS: docker.io/opensearchproject/opensearch:2.4.0
Version 2.4.0

The text was updated successfully, but these errors were encountered:

anasalkouz · 2022-11-22T21:14:33Z

@davidlago Could you look into this? Shall we transfer this to security plugin repo?

dblock · 2022-11-23T15:55:15Z

@anasalkouz Doesn't look like a security plugin problem.

This plugin hasn't been touched in a long time.

@max-frank I see that the stack says Caused by: java.lang.SecurityException: access denied ("java.net.SocketPermission" "metadata.google.internal:80" "connect,resolve"). GCE hosts have access to metadata.google.internal without auth, so assuming there's no issue hitting that from the host, you've correctly identified that this is a java permission issue. Usually this is fixed by adding java.net.SocketPermission to the host in the java.policy file (see SO). Do you have any changes in plugin-security.policy? What do you have with java.net.SocketPermission in your instance? I see that in our code we say "connect", but not "resolve", maybe that's a bug? Try adding it?

I see @willyborankin contribute to this a while ago, maybe you can help?

willyborankin · 2022-11-23T16:15:43Z

@dblock I added SOCKS5 proxy support for GCS and added this new permission:

// gcs client set Authenticator for proxy username/password
permission java.net.NetPermission "setDefaultAuthenticator";

regarding java.net.SocketPermission" "metadata.google.internal:80" "connect" it was added 6 yaers ago, so it could be that it needs to be extended to connect,resolve at least we use our own solution with such settings in prod env and it works just fine for OS 2.3. I can prepare a fix for it

dblock · 2022-11-23T16:36:54Z

@willyborankin Thanks for jumping in!

I think we should let @max-frank confirm that java.net.SocketPermission" "metadata.google.internal:80" "connect,resolve" fixes his problem first. Does it? If it does, appreciate a PR from you, @max-frank, please?

For proxy support, I believe this may be necessary, but I don't know if there are consequences of having permission java.net.NetPermission "setDefaultAuthenticator"; by default. If you can't think of any, we should add it in the defaults too. we already have this

willyborankin · 2022-11-23T16:38:38Z

@dblock Sure no problem. BTW @max-frank what JDK do you use?

max-frank · 2022-11-24T00:37:04Z

@willyborankin @dblock Thanks for the quick replies. Let give me a bit to check if I can make it work by adding the suggested permissions. I added the additional info about JDK and current policy below:

JDK is:

$  jdk/bin/java --version
openjdk 17.0.5 2022-10-18
OpenJDK Runtime Environment Temurin-17.0.5+8 (build 17.0.5+8)
OpenJDK 64-Bit Server VM Temurin-17.0.5+8 (build 17.0.5+8, mixed mode, sharing)

which should be default for the official container image i.e.,

{
  "image": "docker.io/opensearchproject/opensearch:2.4.0",
  "imageID": "docker.io/opensearchproject/opensearch@sha256:ffd7da5e9b8365ce49596f8bd66b782fe99321db57badb263c29a21ec4e4697c"
}

My plugin-security.policy is unchanged from the defaults so:

$ sha256sum plugins/repository-gcs/plugin-security.policy 
51ced1a5ff9d6fd5d2cd45beb68a6290e35a5dae2245c21c2390774ab765b197  plugins/repository-gcs/plugin-security.policy

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

/*
 * Modifications Copyright OpenSearch Contributors. See
 * GitHub history for details.
 */

grant {
  // required by: com.google.api.client.json.JsonParser#parseValue
  permission java.lang.RuntimePermission "accessDeclaredMembers";
  // required by: com.google.api.client.json.GenericJson#<init>
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
  // required to add google certs to the gcs client trustore
  permission java.lang.RuntimePermission "setFactory";

  // gcs client opens socket connections for to access repository
  permission java.net.SocketPermission "*", "connect";

  // gcs client set Authenticator for proxy username/password
  permission java.net.NetPermission "setDefaultAuthenticator";
};

reta · 2022-11-24T01:01:37Z

It seems like permission java.net.SocketPermission "*", "connect"; should be permission java.net.SocketPermission "*", "connect,resolve";?

max-frank · 2022-11-24T02:27:20Z

Ok so it seems that permission java.net.SocketPermission "*", "connect,resolve"; is not necessary. I have tested with fresh clusters with the exact same deployment configuration and in both cases it worked fine. It seems the problem was that somehow the repository config on the original cluster were I experienced this issue got corrupted resulting in these issues.

Note sure how it got corrupted since the same repository config works fine now after deleting the repositories once and re-adding it with the original configuration. Only thing I can imagine that something went wrong during the cluster upgrade.

Though note that it seems the resolve permissions is detected as required during plugin install anyway even if its not explicitly part of the plugin-security.policy

Defaulted container "opensearch" out of: opensearch, init (init), init-sysctl (init)
-> Installing repository-gcs
-> Downloading repository-gcs from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission setFactory
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve

Sorry for blowing up this issue the original issue is resolved though I have not been able to reproduce the cause yet.

dblock · 2022-11-28T17:53:18Z

Thanks for bringing this up and working through it with us @max-frank!

max-frank added bug Something isn't working untriaged labels Nov 22, 2022

minalsha added the distributed framework label Nov 22, 2022

max-frank closed this as completed Nov 24, 2022

anasalkouz removed the untriaged label Oct 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Getting security exception due to access denied "java" #5331

[BUG] Getting security exception due to access denied "java" #5331

max-frank commented Nov 22, 2022

anasalkouz commented Nov 22, 2022

dblock commented Nov 23, 2022 •

edited

Loading

willyborankin commented Nov 23, 2022

dblock commented Nov 23, 2022 •

edited

Loading

willyborankin commented Nov 23, 2022

max-frank commented Nov 24, 2022

reta commented Nov 24, 2022

max-frank commented Nov 24, 2022

dblock commented Nov 28, 2022

[BUG] Getting security exception due to access denied "java" #5331

[BUG] Getting security exception due to access denied "java" #5331

Comments

max-frank commented Nov 22, 2022

anasalkouz commented Nov 22, 2022

dblock commented Nov 23, 2022 • edited Loading

willyborankin commented Nov 23, 2022

dblock commented Nov 23, 2022 • edited Loading

willyborankin commented Nov 23, 2022

max-frank commented Nov 24, 2022

reta commented Nov 24, 2022

max-frank commented Nov 24, 2022

dblock commented Nov 28, 2022

dblock commented Nov 23, 2022 •

edited

Loading

dblock commented Nov 23, 2022 •

edited

Loading