Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ClassCastException with non-async Servlet + async Filter + HttpServletRequestWrapper #4141

Closed
davidkarlsen opened this issue Oct 1, 2019 · 16 comments
Closed

ClassCastException with non-async Servlet + async Filter + HttpServletRequestWrapper #4141

davidkarlsen opened this issue Oct 1, 2019 · 16 comments
Assignees
@joakime
Labels
Bug For general bugs on Jetty side High Priority

Comments

@davidkarlsen
Copy link

davidkarlsen commented Oct 1, 2019 •
edited
Loading

After upgrading jetty from 9.4.20 -> 9.4.21
I am seeing:

java.lang.ClassCastException: class org.springframework.security.web.firewall.StrictHttpFirewall$1 cannot be cast 
  to class org.eclipse.jetty.server.Request (org.springframework.security.web.firewall.StrictHttpFirewall$1 is in 
  unnamed module of loader org.eclipse.jetty.webapp.WebAppClassLoader @11e355ca; 
  org.eclipse.jetty.server.Request is in unnamed module of loader org.eclipse.jetty.start.Classpath$Loader @311d617d)

in both cases I am running java 12 + spring security 5.1.6
@davidkarlsen davidkarlsen mentioned this issue Oct 1, 2019
Closed
@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

Are you running with Java 11 and JPMS / Modules mode?

@davidkarlsen
Copy link
Author

@joakime no - plain unpack of http://repo1.maven.org/maven2/org/eclipse/jetty/jetty-home/

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

plain unpack of jetty-home is run-less. it has no start options configured.
can you run the following and report back?

$ cd /path/to/my-jetty.base
$ java -jar /path/to/jetty-home/start.jar --list-config

@davidkarlsen
Copy link
Author

docker exec -ti musing_knuth java -jar /app/start.jar --list-config:


Java Environment:
-----------------
 java.home = /usr/lib/jvm/jdk-12.0.2+10 (null)
 java.vm.vendor = AdoptOpenJDK (null)
 java.vm.version = 12.0.2+10 (null)
 java.vm.name = OpenJDK 64-Bit Server VM (null)
 java.vm.info = mixed mode, sharing (null)
 java.runtime.name = OpenJDK Runtime Environment (null)
 java.runtime.version = 12.0.2+10 (null)
 java.io.tmpdir = /tmp (null)
 user.dir = /app (null)
 user.language = en (null)
 user.country = null (null)

Jetty Environment:
-----------------
 jetty.version = 9.4.21.v20190926
 jetty.tag.version = master
 jetty.home = /app
 jetty.base = /app

Config Search Order:
--------------------
 <command-line>
 ${jetty.base} -> /app
 ${jetty.home} -> /app


JVM Arguments:
--------------
 (no jvm args specified)

System Properties:
------------------
 (no system properties specified)

Properties:
-----------
 java.version = 12.0.2
 java.version.major = 12
 java.version.micro = 2
 java.version.minor = 0
 java.version.platform = 12
 jetty.base = /app
 jetty.base.uri = file:///app
 jetty.home = /app
 jetty.home.uri = file:///app
 jetty.webapp.addServerClasses = -org.eclipse.jetty.servlet.StatisticsServlet

Jetty Server Classpath:
-----------------------
Version Information on 28 entries in the classpath.
Note: order presented here is how they would appear on the classpath.
      changes to the --module=name command line options will be reflected here.
 0:      1.4.1.v201005082020 | ${jetty.base}/lib/mail/javax.mail.glassfish-1.4.1.v201005082020.jar
 1:                    3.1.0 | ${jetty.base}/lib/servlet-api-3.1.jar
 2:                 3.1.0.M0 | ${jetty.base}/lib/jetty-schemas-3.1.jar
 3:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-http-9.4.21.v20190926.jar
 4:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-server-9.4.21.v20190926.jar
 5:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-xml-9.4.21.v20190926.jar
 6:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-util-9.4.21.v20190926.jar
 7:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-io-9.4.21.v20190926.jar
 8:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-jndi-9.4.21.v20190926.jar
 9:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-security-9.4.21.v20190926.jar
10:                      1.3 | ${jetty.base}/lib/transactions/javax.transaction-api-1.3.jar
11:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-servlet-9.4.21.v20190926.jar
12:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-webapp-9.4.21.v20190926.jar
13:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-plus-9.4.21.v20190926.jar
14:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-annotations-9.4.21.v20190926.jar
15:                      7.1 | ${jetty.base}/lib/annotations/asm-7.1.jar
16:                      7.1 | ${jetty.base}/lib/annotations/asm-analysis-7.1.jar
17:                      7.1 | ${jetty.base}/lib/annotations/asm-commons-7.1.jar
18:                      7.1 | ${jetty.base}/lib/annotations/asm-tree-7.1.jar
19:                      1.3 | ${jetty.base}/lib/annotations/javax.annotation-api-1.3.jar
20:    3.17.0.v20190306-2240 | ${jetty.base}/lib/apache-jsp/org.eclipse.jdt.ecj-3.17.0.jar
21:         9.4.21.v20190926 | ${jetty.base}/lib/apache-jsp/org.eclipse.jetty.apache-jsp-9.4.21.v20190926.jar
22:                   8.5.40 | ${jetty.base}/lib/apache-jsp/org.mortbay.jasper.apache-el-8.5.40.jar
23:                   8.5.40 | ${jetty.base}/lib/apache-jsp/org.mortbay.jasper.apache-jsp-8.5.40.jar
24:                    1.2.5 | ${jetty.base}/lib/apache-jstl/org.apache.taglibs.taglibs-standard-impl-1.2.5.jar
25:                    1.2.5 | ${jetty.base}/lib/apache-jstl/org.apache.taglibs.taglibs-standard-spec-1.2.5.jar
26:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-deploy-9.4.21.v20190926.jar
27:         9.4.21.v20190926 | ${jetty.base}/lib/jetty-jmx-9.4.21.v20190926.jar

Jetty Active XMLs:
------------------
 ${jetty.base}/etc/jetty-bytebufferpool.xml
 ${jetty.base}/etc/jetty-threadpool.xml
 ${jetty.base}/etc/jetty.xml
 ${jetty.base}/etc/jetty-webapp.xml
 ${jetty.base}/etc/jetty-plus.xml
 ${jetty.base}/etc/jetty-annotations.xml
 ${jetty.base}/etc/jetty-deploy.xml
 ${jetty.base}/etc/jetty-http.xml
 ${jetty.base}/etc/jetty-jmx.xml
 ${jetty.base}/etc/jetty-jmx-remote.xml
 ${jetty.base}/etc/jetty-requestlog.xml
 ${jetty.base}/etc/jetty-stats.xml

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

No red flags in that output.

But you do have a yellow flag.
Your jetty.base and jetty.home directory are the same directory, this is not recommended.

Create a separate jetty.base directory from your jetty.home directory.
The 2 directories should not be nested within each other in either direction.

I see you are using docker, you should continue to keep jetty.home and jetty.base separate.

The official docker image at https:/appropriate/docker-jetty/blob/master/9.4-jre11/Dockerfile
Uses /usr/local/jetty for JETTY_HOME
and /var/lib/jetty for JETTY_BASE

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

Do you have a demo webapp that can replicate this?

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

Also, do you have a longer stacktrace for that error?

@davidkarlsen
Copy link
Author

Does it really matter with separate home&base? I'll only run a single app anyways - and it has worked for many years until 9.4.21.

Unfortunately it is closed source.

Full stack:

2019-10-01 20:49:05,184 [main][][][][][][][] INFO com.edb.fs.tac.jfr.srv.Application - Started Application in 47.509 seconds (JVM running for 72.169)
2019-10-01 20:49:05,207 [main][][][][][][][] INFO org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2019-10-01 20:49:05,233 [main][][][][][][][] INFO org.springframework.web.servlet.DispatcherServlet - Completed initialization in 25 ms
2019-10-01 20:49:05,709 [qtp1836797772-15][][][][][][][] ERROR org.springframework.boot.web.servlet.support.ErrorPageFilter - Forwarding to error page from request [/index.jsp] due to exception [class org.springframework.security.web.firewall.StrictHttpFirewall$1 cannot be cast to class org.eclipse.jetty.server.Request (org.springframework.security.web.firewall.StrictHttpFirewall$1 is in unnamed module of loader org.eclipse.jetty.webapp.WebAppClassLoader @11e355ca; org.eclipse.jetty.server.Request is in unnamed module of loader org.eclipse.jetty.start.Classpath$Loader @311d617d)]
java.lang.ClassCastException: class org.springframework.security.web.firewall.StrictHttpFirewall$1 cannot be cast to class org.eclipse.jetty.server.Request (org.springframework.security.web.firewall.StrictHttpFirewall$1 is in unnamed module of loader org.eclipse.jetty.webapp.WebAppClassLoader @11e355ca; org.eclipse.jetty.server.Request is in unnamed module of loader org.eclipse.jetty.start.Classpath$Loader @311d617d)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at ch.qos.logback.classic.helpers.MDCInsertingServletFilter.doFilter(MDCInsertingServletFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:128)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:66)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:103)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:121)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:536)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1589)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1296)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1559)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1211)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:560)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:268)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:782)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:914)
	at java.base/java.lang.Thread.run(Thread.java:835)

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

Do you have any jetty jars in your webapp's WEB-INF/lib? If so, which ones?

@davidkarlsen
Copy link
Author

Nope:

find . -name \*jetty\*jar
./jetty-0_0_0_0-8080-ccm-finodsping-dist-4_0_4_war-_ccm-finodsping-dist-4_0_4-any-536137581480786538.dir/webapp/WEB-INF/lib/simpleclient_jetty_jdk8-0.7.0.jar
./jetty-0_0_0_0-8080-ccm-finodsping-dist-4_0_4_war-_ccm-finodsping-dist-4_0_4-any-536137581480786538.dir/webapp/WEB-INF/lib/simpleclient_jetty-0.7.0.jar

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019 •
edited
Loading

So Spring Security has a HttpServletRequestWrapper it calls FirewalledRequest.

That abstract class is being defined as an anonymous class within Spring Security's StrictHttpFirewall

When the Jetty NotAsyncServlet kicks in, the unwrapping to jetty's Request object fails.

https:/eclipse/jetty.project/blob/72970db61a2904371e1218a95a3bef5d79788c33/jetty-servlet/src/main/java/org/eclipse/jetty/servlet/ServletHolder.java#L1384-L1391

The only way I can replicate your java.lang.ClassCastException stacktrace is by having the jetty-server.jar in the WEB-INF/lib of the webapp.

Same jetty-home version.
Same java version.
Same jetty.base configuration.

Going to try a few other things to attempt to replicate.

@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

There we go!

I was able to replicate in a test case.

2019-10-01 17:20:17.787:WARN:oejs.HttpChannel:qtp183284570-49: /hello
java.lang.ClassCastException: class org.eclipse.jetty.servlet.ServletWrapperTest$NoopRequestWrapper cannot be cast to class org.eclipse.jetty.server.Request (org.eclipse.jetty.servlet.ServletWrapperTest$NoopRequestWrapper and org.eclipse.jetty.server.Request are in unnamed module of loader 'app')
	at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at org.eclipse.jetty.servlet.ServletWrapperTest$WrapFilter.doFilter(ServletWrapperTest.java:116)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)

It triggers when all of the following is true.

  1. the Servlet endpoint is not-async.
  2. the Filter is async.
  3. the HttpServletRequest is wrapped.

It's an easy enough fix. (and now we have a testcase for it)

@joakime joakime added the Bug For general bugs on Jetty side label Oct 1, 2019
@joakime joakime self-assigned this Oct 1, 2019
@joakime
Copy link
Contributor

joakime commented Oct 1, 2019

Opened PR #4143

@gregw
Copy link
Contributor

gregw commented Oct 1, 2019

@davidkarlsen if you need a work around for this problem in 9.4.21, you can simple define a noop filter that applies only to your servlet and mark the filter as not-async. This will avoid the bad case in ServletHolder.

@joakime joakime mentioned this issue Oct 1, 2019
Closed
@davidkarlsen
Copy link
Author

Thanks! That was fast 😊

@joakime joakime changed the title Woes with jetty and spring-security after upgrading from 9.4.20 -> 9.4.21 ClassCastException with Servlet non-async + async Filter + HttpServletRequestWrapper Oct 2, 2019
@joakime joakime changed the title ClassCastException with Servlet non-async + async Filter + HttpServletRequestWrapper ClassCastException with non-async Servlet + async Filter + HttpServletRequestWrapper Oct 2, 2019
@joakime joakime closed this as completed in 0a5c710 Oct 2, 2019
joakime added a commit that referenced this issue Oct 2, 2019
@joakime
Merge pull request #4143 from eclipse/jetty-9.4.x-4141-classcastexcep…
ed9031b 
…tion-request-async

Fixes #4141 - Servlet not async + async Filter + wrapped request
@joakime
Copy link
Contributor

joakime commented Oct 30, 2019

Fixed in Jetty 9.4.22.v20191022

@joakime joakime mentioned this issue Oct 30, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joakime joakime

Labels
Bug For general bugs on Jetty side High Priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@davidkarlsen @gregw @joakime