Merge pull request #336 from newrelic/release/v1.5.0

CSEC Java Agent Public Release Version 1.5.0

Changelog.md

@@ -4,6 +4,95 @@ Noteworthy changes to the agent are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.5.0] - 2024-9-25
+### New features
+- Json Version bump to 1.2.9.
+- [PR-327](https:/newrelic/csec-java-agent/pull/327) Application endpoint detection for gRPC Server [NR-303616](https://new-relic.atlassian.net/browse/NR-303616)
+- [PR-326](https:/newrelic/csec-java-agent/pull/326) Add IAST Scan start time and Traffic Start Time in Health Check [NR-308822](https://new-relic.atlassian.net/browse/NR-308822)
+- [PR-320](https:/newrelic/csec-java-agent/pull/320) Add feature to allow IAST Scan Scheduling. [NR-301534](https://new-relic.atlassian.net/browse/NR-301534)
+ Configuration via yaml:
+ ```yaml
+ security:
+ scan_schedule:
+ # The delay field specifies the delay in minutes before the IAST scan starts. This allows to schedule the scan to start at a later time.
+ delay: 0 #In minutes, default is 0 min
+
+ # The duration field specifies the duration of the IAST scan in minutes. This determines how long the scan will run.
+ duration: 0 #In minutes, default is forever
+
+ # The schedule field specifies a cron expression that defines when the IAST scan should start.
+ #schedule: "" #By default, schedule is inactive
+
+ # Allow continuously sample collection of IAST events
+ always_sample_traces: false # Default is false
+ ```
+- [PR-320](https:/newrelic/csec-java-agent/pull/320) Add feature to ignore IAST Scan of certain APIs, categories, or parameters. [NR-301856](https://new-relic.atlassian.net/browse/NR-301856)
+ Configuration via yaml:
+ ```yaml
+ security:
+ # The exclude_from_iast_scan configuration allows to specify APIs, parameters, and categories that should not be scanned by Security Agents.
+ exclude_from_iast_scan:
+ # The api field specifies list of APIs using regular expression (regex) patterns that follow the syntax of Perl 5. The regex pattern should provide a complete match for the URL without the endpoint.
+ # Example:
+ # api:
+ # - .*account.*
+ # - .*/\api\/v1\/.*?\/login
+ api: []
+
+ # The parameters configuration allows users to specify headers, query parameters, and body keys that should be excluded from IAST scans.
+ # Example:
+ # http_request_parameters:
+ # header:
+ # - X-Forwarded-For
+ # query:
+ # - username
+ # - password
+ # body:
+ # - account.email
+ # - account.contact
+ http_request_parameters:
+ # A list of HTTP header keys. If a request includes any headers with these keys, the corresponding IAST scan will be skipped.
+ header: []
+ # A list of query parameter keys. The presence of these parameters in the request's query string will lead to skipping the IAST scan.
+ query: []
+ # A list of keys within the request body. If these keys are found in the body content, the IAST scan will be omitted.
+ body: []
+
+ # The iast_detection_category configuration allows to specify which categories of vulnerabilities should not be detected by Security Agents.
+ # If any of these categories are set to true, Security Agents will not generate events or flag vulnerabilities for that category.
+ iast_detection_category:
+ insecure_settings: false
+ invalid_file_access: false
+ sql_injection: false
+ nosql_injection: false
+ ldap_injection: false
+ javascript_injection: false
+ command_injection: false
+ xpath_injection: false
+ ssrf: false
+ rxss: false
+ ```
+- [PR-321](https:/newrelic/csec-java-agent/pull/321) Add feature to rate limit the IAST replay requests. [NR-304574](https://new-relic.atlassian.net/browse/NR-304574)
+ ```yaml
+ security:
+ scan_controllers:
+ # The scan_request_rate_limit configuration allows to specify maximum number of replay request played per minute.
+ iast_scan_request_rate_limit: 3600 # Number of IAST replay request played per minute, Default is 3600
+ ```
+- [PR-315](https:/newrelic/csec-java-agent/pull/315) GraphQL Support : The security agent now also supports GraphQL Version 16.0.0 and above, default is disabled. [NR-299885](https://new-relic.atlassian.net/browse/NR-299885)
+
+### Fixes
+- [PR-322](https:/newrelic/csec-java-agent/pull/322) Report Application endpoints immediately upon detecting new endpoints. [NR-287324](https://new-relic.atlassian.net/browse/NR-287324)
+- [PR-323](https:/newrelic/csec-java-agent/pull/323) Extract Server Configuration to resolve IAST localhost connection with application for WebSphere Liberty server [NR-303483](https://new-relic.atlassian.net/browse/NR-303483)
+- [PR-327](https:/newrelic/csec-java-agent/pull/327) Fix for User Class Detection in gRPC Server [NR-303616](https://new-relic.atlassian.net/browse/NR-303616)
+- [PR-328](https:/newrelic/csec-java-agent/pull/328) Fix for multiple Reflected Events observed in Jersey Framework [NR-307644](https://new-relic.atlassian.net/browse/NR-307644)
+- [PR-325](https:/newrelic/csec-java-agent/pull/325) Fix for incorrect Application endpoints detected for Servlet Framework [NR-303615](https://new-relic.atlassian.net/browse/NR-303615)
+- [PR-320](https:/newrelic/csec-java-agent/pull/320) Report only uncaught exceptions in IAST Error inbox. [NR-313412](https://new-relic.atlassian.net/browse/NR-313412)
+
+### Deprecations
+- Status File Used for Debugging: This feature has been deprecated. All debugging capabilities have been moved to either Init Logging or [Error Inbox](https://docs.newrelic.com/docs/errors-inbox/errors-inbox/) and will be removed in a future agent release. [NR-293966](https://new-relic.atlassian.net/browse/NR-293966)
+
+
 ## [1.4.1] - 2024-8-14
 ### Adds
 - [PR-296](https:/newrelic/csec-java-agent/pull/296) Apache Solr Support: The security agent now also supports Apache Solr Version 4.0.0 and above. [NR-288599](https://new-relic.atlassian.net/browse/NR-288599)

README.md

@@ -33,7 +33,7 @@ The agent automatically instruments the following frameworks.
 - Resin 3.1.9 to 4.0.x
 - Jetty 9.3.0.M1 to latest
 - Mule ESB 3.6 to 3.9.x
-- gRPC 1.4.0 to latest**
+- gRPC 1.4.0 to latest [**](#grpc-instrumentation)
 - Jersey 2.0 to latest
 - Akka Server 10.0 to latest (with scala 2.11 and above)
 - Spray Can 1.3.1 to latest (with scala 2.11 and above)
@@ -42,8 +42,18 @@ The agent automatically instruments the following frameworks.
 - Netty Server 4.0.0.Final to latest
 - Netty Reactor Server 0.7.0.RELEASE to latest
 - Vertx web 3.2.0 to latest
-
-** IAST for **gRPC** requires the dependency [protobuf-java-util](https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util) for IAST request replay.
+- GraphQL 16.0.0 to latest [**](#graphql-instrumentation)
+
+#### gRPC Instrumentation
+IAST for **gRPC** requires the dependency [protobuf-java-util](https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util) for IAST request replay.
+
+#### GraphQL Instrumentation
+By default, GraphQL instrumentation is disabled in IAST as it is an experimental feature. To take advantage of this feature enable GraphQL instrumentation, update your configuration by adding the following settings under the class_transformer section:
+```yaml
+class_transformer:
+ com.newrelic.instrumentation.security.graphql-java-16.2:
+ enabled: true
+```
 
 ### Java Native Operations
 

gradle.properties

@@ -1,5 +1,5 @@
 # The agent version.
-agentVersion=1.4.2
+agentVersion=1.5.0
 jsonVersion=1.2.9
 # Updated exposed NR APM API version.
 nrAPIVersion=8.12.0

instrumentation-security/graalvm-jsinjection-22.0.0/build.gradle

@@ -13,6 +13,7 @@ jar {
 
 verifyInstrumentation {
  passesOnly 'org.graalvm.js:js:[22.0.0,)'
+ exclude('org.graalvm.js:js:24.1.0')
 // excludeRegex '.*-rc[0-9]+'
 }
 

instrumentation-security/graphql-java-16.2/build.gradle

@@ -6,7 +6,7 @@ dependencies {
 }
 
 jar {
- manifest { attributes 'Implementation-Title': 'com.newrelic.instrumentation.security.graphql-java-16.2' }
+ manifest { attributes 'Implementation-Title': 'com.newrelic.instrumentation.security.graphql-java-16.2', 'Enabled': 'false' }
 }
 
 verifyInstrumentation {

instrumentation-security/jedis-1.4.0/build.gradle

@@ -7,7 +7,7 @@ dependencies {
  implementation("com.newrelic.agent.java:newrelic-api:${nrAPIVersion}")
  implementation("com.newrelic.agent.java:newrelic-weaver-api:${nrAPIVersion}")
  implementation("redis.clients:jedis:1.4.0")
- testImplementation('org.testcontainers:testcontainers:1.17.1')
+ testImplementation('org.testcontainers:testcontainers:1.20.1')
 }
 
 verifyInstrumentation {

instrumentation-security/jedis-2.7.1_2.7.2/build.gradle

@@ -7,7 +7,7 @@ dependencies {
  implementation("com.newrelic.agent.java:newrelic-api:${nrAPIVersion}")
  implementation("com.newrelic.agent.java:newrelic-weaver-api:${nrAPIVersion}")
  implementation("redis.clients:jedis:2.7.1")
- testImplementation('org.testcontainers:testcontainers:1.17.1')
+ testImplementation('org.testcontainers:testcontainers:1.20.1')
 }
 
 verifyInstrumentation {

instrumentation-security/jedis-3.0.0/build.gradle

@@ -7,7 +7,7 @@ dependencies {
  implementation("com.newrelic.agent.java:newrelic-api:${nrAPIVersion}")
  implementation("com.newrelic.agent.java:newrelic-weaver-api:${nrAPIVersion}")
  implementation("redis.clients:jedis:3.0.0")
- testImplementation('org.testcontainers:testcontainers:1.17.1')
+ testImplementation('org.testcontainers:testcontainers:1.20.1')
 }
 
 verifyInstrumentation {

instrumentation-security/jedis-4.0.0/build.gradle

@@ -7,7 +7,7 @@ dependencies {
  implementation("com.newrelic.agent.java:newrelic-api:${nrAPIVersion}")
  implementation("com.newrelic.agent.java:newrelic-weaver-api:${nrAPIVersion}")
  implementation("redis.clients:jedis:4.0.0")
- testImplementation('org.testcontainers:testcontainers:1.17.1')
+ testImplementation('org.testcontainers:testcontainers:1.20.1')
 }
 
 verifyInstrumentation {

instrumentation-security/jersey-2.16/src/main/java/com/newrelic/agent/security/instrumentation/jersey2/ApplicationHandler_Handler.java

@@ -7,14 +7,12 @@
 
 package com.newrelic.agent.security.instrumentation.jersey2;
 
-import com.newrelic.api.agent.Trace;
+import com.newrelic.api.agent.security.instrumentation.helpers.GenericHelper;
 import com.newrelic.api.agent.weaver.MatchType;
 import com.newrelic.api.agent.weaver.Weave;
 import com.newrelic.api.agent.weaver.Weaver;
 import org.glassfish.jersey.server.ContainerRequest;
 
-import java.io.OutputStream;
-
 @Weave(type = MatchType.ExactClass, originalName = "org.glassfish.jersey.server.ApplicationHandler")
 public abstract class ApplicationHandler_Handler {
 
@@ -24,6 +22,7 @@ public void handle(ContainerRequest requestContext) {
  if (requestContext != null) {
  isRequestLockAcquired = HttpRequestHelper.acquireRequestLockIfPossible();
  if (isRequestLockAcquired) {
+ GenericHelper.acquireLockIfPossible(HttpRequestHelper.getNrSecCustomAttribForPostProcessing());
  HttpRequestHelper.preprocessSecurityHook(requestContext);
  HttpRequestHelper.registerUserLevelCode("JERSEY");
  }
@@ -32,6 +31,7 @@ public void handle(ContainerRequest requestContext) {
  } finally {
  if(isRequestLockAcquired){
  HttpRequestHelper.releaseRequestLock();
+ GenericHelper.releaseLock(HttpRequestHelper.getNrSecCustomAttribForPostProcessing());
  }
  }
  }

instrumentation-security/jersey-2.16/src/main/java/com/newrelic/agent/security/instrumentation/jersey2/ContainerResponse_Instrumentation.java

@@ -19,9 +19,6 @@
 import org.glassfish.jersey.message.internal.OutboundMessageContext;
 import org.glassfish.jersey.server.ContainerRequest;
 
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
 
 import static com.newrelic.api.agent.security.instrumentation.helpers.ServletHelper.SERVLET_GET_IS_OPERATION_LOCK;
 
@@ -33,7 +30,7 @@ public abstract class ContainerResponse_Instrumentation {
  NewRelicSecurity.getAgent().getSecurityMetaData().getResponse().setResponseCode(response.getStatus());
  }
 
- if(response != null && response.getContext() != null && response.getContext().hasEntity()){
+ if(GenericHelper.isLockAcquired(HttpRequestHelper.getNrSecCustomAttribForPostProcessing()) && response != null && response.getContext() != null && response.getContext().hasEntity()){
  Object responseObject = response.getContext().getEntity();
  NewRelicSecurity.getAgent().getSecurityMetaData().getResponse().setResponseBody(new StringBuilder(String.valueOf(responseObject)));
  }
@@ -45,7 +42,7 @@ public void close() {
  boolean isLockAcquired = false;
  try {
  isLockAcquired = GenericHelper.acquireLockIfPossible(VulnerabilityCaseType.REFLECTED_XSS, SERVLET_GET_IS_OPERATION_LOCK);
- if(isLockAcquired) {
+ if(isLockAcquired && GenericHelper.isLockAcquired(HttpRequestHelper.getNrSecCustomAttribForPostProcessing())) {
  HttpRequestHelper.postProcessSecurityHook(this.getClass().getName(), getWrappedMessageContext());
  }
  Weaver.callOriginal();

instrumentation-security/jersey-2.16/src/main/java/com/newrelic/agent/security/instrumentation/jersey2/HttpRequestHelper.java

@@ -13,7 +13,6 @@
 import com.newrelic.api.agent.security.schema.operation.RXSSOperation;
 import com.newrelic.api.agent.security.schema.policy.AgentPolicy;
 import com.newrelic.api.agent.security.utils.logging.LogLevel;
-import jdk.nashorn.internal.codegen.CompilerConstants;
 import org.glassfish.jersey.internal.PropertiesDelegate;
 import org.glassfish.jersey.message.internal.OutboundMessageContext;
 import org.glassfish.jersey.server.ContainerRequest;
@@ -28,36 +27,24 @@ public class HttpRequestHelper {
 
  private static final String X_FORWARDED_FOR = "x-forwarded-for";
  private static final String EMPTY = "";
- public static final String QUESTION_MARK = "?";
  public static final String CONTAINER_RESPONSE_METHOD_NAME = "ContainerResponse";
 
- public static final String NR_SEC_CUSTOM_ATTRIB_NAME = "REQUEST_LOCK-";
- private static final String WILDCARD = "*";
- private static final String SEPARATOR = "/";
+ public static final String NR_SEC_CUSTOM_ATTRIB_NAME = "SERVLET_LOCK-";
+ private static final String NR_SEC_CUSTOM_ATTRIB_NAME_POST_PROCESSING = "JERSEY_LOCK_POST_PROCESSING-";
  public static final String HEADER_SEPARATOR = ";";
- public static final String GRIZZLY_REQUEST_PROPERTIES_DELEGATE = "GRIZZLY_REQUEST_PROPERTIES_DELEGATE";
- public static final String GRIZZLY_REQUEST = "GRIZZLY_REQUEST";
  public static final String ORG_GLASSFISH_JERSEY_GRIZZLY_2_HTTPSERVER_GRIZZLY_REQUEST_PROPERTIES_DELEGATE = "org.glassfish.jersey.grizzly2.httpserver.GrizzlyRequestPropertiesDelegate";
- public static final String ORG_GLASSFISH_GRIZZLY_HTTP_SERVER_REQUEST = "org.glassfish.grizzly.http.server.Request";
  public static final String FIELD_REQUEST = "request";
  public static final String METHOD_GET_REMOTE_ADDR = "getRemoteAddr";
  public static final String METHOD_GET_REMOTE_PORT = "getRemotePort";
  public static final String METHOD_GET_LOCAL_PORT = "getLocalPort";
  public static final String METHOD_GET_SCHEME = "getScheme";
  public static final String METHOD_GET_CONTENT_TYPE = "getContentType";
  public static final String ORG_GLASSFISH_JERSEY_GRIZZLY_2_HTTPSERVER_TRACING_AWARE_PROPERTIES_DELEGATE = "org.glassfish.jersey.message.internal.TracingAwarePropertiesDelegate";
- public static final String TRACING_AWARE_PROPERTIES_DELEGATE = "TRACING_AWARE_PROPERTIES_DELEGATE";
  public static final String FIELD_PROPERTIES_DELEGATE = "propertiesDelegate";
 
  private static final String REQUEST_INPUTSTREAM_HASH = "REQUEST_INPUTSTREAM_HASH";
  public static final String JERSEY_2_16 = "JERSEY-2.16";
 
- public static Class grizzlyRequestPropertiesDelegateKlass = null;
-
- public static Class grizzlyRequest = null;
-
- public static Class tracingAwarePropertiesDelegateKlass = null;
-
  public static void preprocessSecurityHook(ContainerRequest requestContext) {
  try {
  if (!NewRelicSecurity.isHookProcessingActive()) {
@@ -91,7 +78,7 @@ public static void preprocessSecurityHook(ContainerRequest requestContext) {
 
  public static void postProcessSecurityHook(String className, OutboundMessageContext wrappedMessageContext) {
  try {
- if (!NewRelicSecurity.isHookProcessingActive()
+ if (!NewRelicSecurity.isHookProcessingActive() || Boolean.TRUE.equals(NewRelicSecurity.getAgent().getSecurityMetaData().getCustomAttribute("RXSS_PROCESSED", Boolean.class))
  ) {
  return;
  }
@@ -197,7 +184,7 @@ public static String getTraceHeader(Map<String, String> headers) {
  return data;
  }
 
- public static boolean isRequestLockAcquired() {
+ private static boolean isRequestLockAcquired() {
  try {
  return NewRelicSecurity.isHookProcessingActive() &&
  Boolean.TRUE.equals(NewRelicSecurity.getAgent().getSecurityMetaData().getCustomAttribute(getNrSecCustomAttribName(), Boolean.class));
@@ -228,20 +215,24 @@ private static String getNrSecCustomAttribName() {
  return NR_SEC_CUSTOM_ATTRIB_NAME + Thread.currentThread().getId();
  }
 
+ public static String getNrSecCustomAttribForPostProcessing() {
+ return NR_SEC_CUSTOM_ATTRIB_NAME_POST_PROCESSING + Thread.currentThread().getId();
+ }
+
  public static void processPropertiesDelegate(PropertiesDelegate propertiesDelegate, HttpRequest securityRequest) {
  if(StringUtils.equals(propertiesDelegate.getClass().getName(), ORG_GLASSFISH_JERSEY_GRIZZLY_2_HTTPSERVER_GRIZZLY_REQUEST_PROPERTIES_DELEGATE)){
  try {
- Class grizzlyRequestPropertiesDelegateKlass = propertiesDelegate.getClass();
+ Class<? extends PropertiesDelegate> grizzlyRequestPropertiesDelegateKlass = propertiesDelegate.getClass();
  Field requestField = grizzlyRequestPropertiesDelegateKlass.getDeclaredField(FIELD_REQUEST);
  requestField.setAccessible(true);
  Object requestObject = requestField.get(propertiesDelegate);
- Class requestClass = requestObject.getClass();
+
+ Class<?> requestClass = requestObject.getClass();
  Method getRemoteAddr = requestClass.getMethod(METHOD_GET_REMOTE_ADDR);
  Method getRemotePort = requestClass.getMethod(METHOD_GET_REMOTE_PORT);
  Method getLocalPort = requestClass.getMethod(METHOD_GET_LOCAL_PORT);
  Method getScheme = requestClass.getMethod(METHOD_GET_SCHEME);
  Method getContentType = requestClass.getMethod(METHOD_GET_CONTENT_TYPE);
- getContentType = requestClass.getMethod(METHOD_GET_CONTENT_TYPE);
  securityRequest.setClientIP(String.valueOf(getRemoteAddr.invoke(requestObject)));
  securityRequest.setClientPort(String.valueOf(getRemotePort.invoke(requestObject)));
  securityRequest.setServerPort((int) getLocalPort.invoke(requestObject));
@@ -255,7 +246,7 @@ public static void processPropertiesDelegate(PropertiesDelegate propertiesDelega
 
  } else if (StringUtils.equals(propertiesDelegate.getClass().getName(), ORG_GLASSFISH_JERSEY_GRIZZLY_2_HTTPSERVER_TRACING_AWARE_PROPERTIES_DELEGATE)){
  try {
- Class tracingAwarePropertiesDelegateKlass = propertiesDelegate.getClass();
+ Class<? extends PropertiesDelegate> tracingAwarePropertiesDelegateKlass = propertiesDelegate.getClass();
  Field propertiesDelegateField = tracingAwarePropertiesDelegateKlass.getDeclaredField(FIELD_PROPERTIES_DELEGATE);
  propertiesDelegateField.setAccessible(true);
  Object propertiesDelegateObject = propertiesDelegateField.get(propertiesDelegate);