Tyrus WebSocket sporadic client side disconnects

[akhodakivskiy]

I'm using Tyrus client to connect to multiple WebSocket servers (around 10 connections) in parallel. The servers publish a lot of messages (hundreds of millions of messages per day). At some point, one of the connections begins to experience client side disconnects. The onClose handlers is invoked with close reason CloseReason[1006,Closed abnormally.]. The onError handler is not invoked. After the error, the connection is re-established using the ReconnectHandler.

Couple of facts regarding those disconnects/reconnects:

• It's always the connection to the same server that has issues. All other connections behave as expected.
• When I estalish the connection to just that one server that is having issues then no disconnects take place
• The first disconnect occurs after about an hour after the start of the application
• Then disconnects happen every few minutes

I also have the stack trace from the onError handler:

2020-11-16 16:23:03,347 3985311 [Grizzly(2)] INFO  MyEndpoint - java.lang.Throwable
        at MyEndpoint.onClose(MyEndpoint.scala:185)
        at org.glassfish.tyrus.core.TyrusEndpointWrapper.onClose(TyrusEndpointWrapper.java:1235)
        at org.glassfish.tyrus.core.TyrusWebSocket.onClose(TyrusWebSocket.java:110)
        at org.glassfish.tyrus.core.ProtocolHandler.close(ProtocolHandler.java:481)
        at org.glassfish.tyrus.core.TyrusWebSocket.close(TyrusWebSocket.java:244)
        at org.glassfish.tyrus.client.TyrusClientEngine$2$1.close(TyrusClientEngine.java:613)
        at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter$CloseTask.execute(GrizzlyClientFilter.java:470)
        at org.glassfish.tyrus.container.grizzly.client.TaskProcessor.processTask(TaskProcessor.java:91)
        at org.glassfish.tyrus.container.grizzly.client.TaskProcessor.processTask(TaskProcessor.java:68)
        at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleClose(GrizzlyClientFilter.java:197)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$4.execute(ExecutorResolver.java:50)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:248)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:181)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:121)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:99)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
        at org.glassfish.grizzly.nio.NIOConnection.preClose(NIOConnection.java:819)
        at org.glassfish.grizzly.nio.transport.TCPNIOConnection.preClose(TCPNIOConnection.java:78)
        at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:557)
        at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:249)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:596)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:59)
        at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:133)
        at org.glassfish.grizzly.ssl.SSLBaseFilter$SSLTransportFilterWrapper.handleRead(SSLBaseFilter.java:919)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:248)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:181)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:121)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:99)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:34)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
        at java.base/java.lang.Thread.run(Thread.java:834)

Here is (roughly) the code I'm using

import jakarta.websocket._

class MyEndpoint extends Endpoint with LazyLogging {
  def onOpen(session: Session, config: EndpointConfig): Unit = {
    session.addMessageHandler(new MessageHandler.Whole[String] {
      def onMessage(message: String): Unit = {
        // handle message
      }
  
      override def onError(session: Session, thr: Throwable): Unit = {
        logger.error(s"error occurred", thr)
  
        super.onError(session, thr)
      }

      override def onClose(session: Session, closeReason: CloseReason): Unit = {
        logger.error(s"closing due to $closeReason")

        logger.info(ExceptionUtils.getStackTrace(new Throwable()))

        super.onClose(session, closeReason)
      }  

    }
  }
}

[Adhira-Deogade]

Interested in the issue.