Handler_ROUTER_HANDLER is leaking memory
Closed this issue · 6 comments
We are using netty-router and seeing memory leak (getting out of memory). It appears from heap dump that Handler_ROUTER_HANDLER is not releasing a buffer. Please find the attached screenshot.
"char[3460] "" Hint: 'io.netty.handler.codec.http.router.Handler_ROUTER_HANDLER' will handle the message from this point.
io.netty.buffer.CompositeByteBuf.touch(CompositeByteBuf.java:1615)
io.netty.buffer.CompositeByteBuf.touch(CompositeByteBuf.java:42)
io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:276)
io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:373)
io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:303)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:156)
io.netty.handler.codec.http.cors.CorsHandler.channelRead(CorsHandler.java:65)
io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1069)
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:944)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:327)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)
io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:946)
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:831)
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:322)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:703)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
java.lang.Thread.run(Thread.java:745)
"char[4096] "" Hint: 'io.netty.handler.codec.http.router.Handler_ROUTED_HANDLER' will handle the message from this point.
io.netty.buffer.DefaultByteBufHolder.touch(DefaultByteBufHolder.java:79)
io.netty.handler.codec.http.websocketx.WebSocketFrame.touch(WebSocketFrame.java:94)
io.netty.handler.codec.http.websocketx.TextWebSocketFrame.touch(TextWebSocketFrame.java:127)
io.netty.handler.codec.http.websocketx.TextWebSocketFrame.touch(TextWebSocketFrame.java:25)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:156)
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
io.netty.handler.codec.http.cors.CorsHandler.channelRead(CorsHandler.java:65)
@ngocdaothanh You can download the heap dump from https://drive.google.com/file/d/0BwPlSk8lfXo9WWZUNm4walFObVk/view?usp=sharing
I haven't found the place that allocates memory but forgets to release memory. Please add this Java option when you run your program:
-Dio.netty.leakDetectionLevel=advanced
With that option, when there's a leak, Netty will show the precise place that allocated memory. See:
http://netty.io/wiki/reference-counted-objects.html#wiki-h2-10
We already use leakDetectionLevel=paranoid but no leak reports yet, it appears from heap dump that most of retained objects are showing
"char[3460] "" Hint: 'io.netty.handler.codec.http.router.Handler_ROUTER_HANDLER' will handle the message from this point.
io.netty.buffer.CompositeByteBuf.touch(CompositeByteBuf.java:1615)
can you please look at heap dump? thanks
What version of Netty are you using?
If you are using Netty 4.1 beta, maybe you should update it.
We use netty-4.1.0-Beta5 (released on 05/08/15). I am waiting for Beta6.
I've just released netty-router 2.0.0.
Please use it (as always, please see README for instructions).
You should be able to download it from Maven in a few minutes.
From this version, netty-router doesn't include any handlers.
It only focus on routing logic.
So you won't have to worry about issue like this.