wttech/CQ-Actions

Many unclosed threads when using CQ Actions sockets

Closed this issue · 0 comments

Current implementation of msg.websocket creates new connection on AEM Author when the old one is disconnected (see SocketClientRunnable run() method). It looks that those connections are not closed properly on AEM Author. After running AEM instance for 2 day I had over 800 threads like this:

Grizzly(1) [DAEMON] State: WAITING CPU usage on sample: 0ms
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:175
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue$Node, LinkedTransferQueue$Node, Object, boolean, long) LinkedTransferQueue.java:737
java.util.concurrent.LinkedTransferQueue.xfer(Object, boolean, int, long) LinkedTransferQueue.java:647
java.util.concurrent.LinkedTransferQueue.take() LinkedTransferQueue.java:1269
org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask() FixedThreadPool.java:105
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork() AbstractThreadPool.java:583
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run() AbstractThreadPool.java:571
java.lang.Thread.run() Thread.java:745

After some time I get:

10.03.2017 14:49:48.820 *ERROR* [FelixStartLevel] ERROR: Bundle '134' EventDispatcher: Error during dispatch. (java.lang.OutOfMemoryError: unable to create new native thread)

The default timeout for connection is 5 minutes and when the connection is idle for that time, connection breaks. A new connection is initiated on AEM Author then.
It looks that there is some problem with proper closing connections that were timed out:

AEM Publish log:

13.03.2017 14:58:19.865 *DEBUG* [qtp774165320-104] com.cognifide.actions.msg.websocket.servlet.MessageSocket Socket Error occured
java.net.SocketTimeoutException: Timeout on Read
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onReadTimeout(AbstractWebSocketConnection.java:545)
        at org.eclipse.jetty.io.AbstractConnection.onFillInterestedFailed(AbstractConnection.java:173)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillInterestedFailed(AbstractWebSocketConnection.java:510)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback$1.run(AbstractConnection.java:586)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)