Many unclosed threads when using CQ Actions sockets
Closed this issue · 0 comments
malaskowski commented
Current implementation of msg.websocket
creates new connection on AEM Author when the old one is disconnected (see SocketClientRunnable
run()
method). It looks that those connections are not closed properly on AEM Author. After running AEM instance for 2 day I had over 800 threads like this:
Grizzly(1) [DAEMON] State: WAITING CPU usage on sample: 0ms
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.park(Object) LockSupport.java:175
java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue$Node, LinkedTransferQueue$Node, Object, boolean, long) LinkedTransferQueue.java:737
java.util.concurrent.LinkedTransferQueue.xfer(Object, boolean, int, long) LinkedTransferQueue.java:647
java.util.concurrent.LinkedTransferQueue.take() LinkedTransferQueue.java:1269
org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask() FixedThreadPool.java:105
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork() AbstractThreadPool.java:583
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run() AbstractThreadPool.java:571
java.lang.Thread.run() Thread.java:745
After some time I get:
10.03.2017 14:49:48.820 *ERROR* [FelixStartLevel] ERROR: Bundle '134' EventDispatcher: Error during dispatch. (java.lang.OutOfMemoryError: unable to create new native thread)
The default timeout for connection is 5 minutes and when the connection is idle for that time, connection breaks. A new connection is initiated on AEM Author then.
It looks that there is some problem with proper closing connections that were timed out:
AEM Publish log:
13.03.2017 14:58:19.865 *DEBUG* [qtp774165320-104] com.cognifide.actions.msg.websocket.servlet.MessageSocket Socket Error occured
java.net.SocketTimeoutException: Timeout on Read
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onReadTimeout(AbstractWebSocketConnection.java:545)
at org.eclipse.jetty.io.AbstractConnection.onFillInterestedFailed(AbstractConnection.java:173)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillInterestedFailed(AbstractWebSocketConnection.java:510)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback$1.run(AbstractConnection.java:586)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)