Tomcat Memory Leaks on netty
patrixgdd opened this issue · 14 comments
hi,
I think this is what I did very similar to this thread:
#29
On the current pushy 0.3, it is Spring 4.0 using Tomcat 7.0.53 with jdk1.7.0_55 x64.
Already put Thread.sleep(2000) after pushManager.shutdown():
INFO: Push manager shutting down.
Apr 21, 2014 11:50:40 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.channel.ChannelHandlerAdapter$1] (value [io.netty.channel.ChannelHandlerAdapter$1@52d67995]) and a value of type [java.util.WeakHashMap] (value [{class com.relayrides.pushy.apns.ApnsConnection$1=true}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 21, 2014 11:50:40 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.util.internal.ThreadLocalRandom$2] (value [io.netty.util.internal.ThreadLocalRandom$2@7ec84faf]) and a value of type [io.netty.util.internal.ThreadLocalRandom] (value [io.netty.util.internal.ThreadLocalRandom@10e7ab30]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 21, 2014 11:50:40 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.util.Recycler$1] (value [io.netty.util.Recycler$1@5997e4fb]) and a value of type [io.netty.util.Recycler.Stack] (value [io.netty.util.Recycler$Stack@33a48b1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Tried several times with the same SEVERE memory leek alerts there.
I thought it could be harmless, but I am not familiar with netty framework. I have to make sure otherwise I need to restart tomcat.
Any suggestions?
I would see if there is a listener to remove the netty from the Tomcat context (eg. on web.xml).
Many Thanks
Hi there!
Netty did introduce thread-local caching in 4.0.18, which we're using in Pushy v0.3. I'll have to dig a little more to understand whether this is actually a leak or just a harmless warning.
I have a hunch that this is related to #74. I see that you've already tried Thread.sleep(2000), but the timeout for the ThreadLocalRandom business is 3 seconds. Would you mind trying again with Thread.sleep(3500) and seeing if that resolves the problem?
Thanks!
Okay,
Tried Thread.sleep(3500) and Thread.sleep(4000).
SERVE memory leaks alert after 2nd restart of the same application.
(1) press STOP on Tomcat Web Application Manager:
Press "find leaks" button
The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):
/xxxapps
On catalina.out:
Apr 23, 2014 11:03:10 PM com.relayrides.pushy.apns.PushManager shutdown
INFO: Push manager shutting down.
(2) Start the application on Tomcat Web Application Manager,
run the application for a while:
Press Press "find leaks" button
The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):
/xxxapps
(3) press STOP on Tomcat Web Application Manager:
Press "find leaks" button
The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):
/xxxapps
/xxxapps
On catalina.out:
Apr 23, 2014 11:08:55 PM com.relayrides.pushy.apns.PushManager shutdown
INFO: Push manager shutting down.
Apr 23, 2014 11:09:01 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.channel.ChannelHandlerAdapter$1] (value [io.netty.channel.ChannelHandlerAdapter$1@5e8c5566]) and a value of type [java.util.WeakHashMap] (value [{class com.relayrides.pushy.apns.ApnsConnection$1=true}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 23, 2014 11:09:01 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.util.internal.ThreadLocalRandom$2] (value [io.netty.util.internal.ThreadLocalRandom$2@4f3acc08]) and a value of type [io.netty.util.internal.ThreadLocalRandom] (value [io.netty.util.internal.ThreadLocalRandom@1d2949db]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 23, 2014 11:09:01 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.util.Recycler$1] (value [io.netty.util.Recycler$1@1afd7d2]) and a value of type [io.netty.util.Recycler.Stack] (value [io.netty.util.Recycler$Stack@3f821cdb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Any suggestions?
@patrixgdd I think this might be related to #74, and might also be fixed by updating to Netty 4.0.19, which we've done in 465777a. Would you mind trying a snapshot from master and seeing if the problem persists?
Thanks!
compiled the pushy 0.4 snapshot, and used the netty 4.0.19-final,
tested with the same application on the same platform using the same steps.
It turns out with exact the same results described above.
okay, more details here
May 07, 2014 10:42:18 PM com.relayrides.pushy.apns.PushManager shutdown
INFO: Push manager shutting down.
May 07, 2014 10:42:25 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/xxxapps] appears to have started a thread named [globalEventExecutor-1-1] but has failed to stop it. This is very likely to create a memory leak.
May 07, 2014 10:42:25 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.util.internal.ThreadLocalRandom$3] (value [io.netty.util.internal.ThreadLocalRandom$3@2da45e90]) and a value of type [io.netty.util.internal.ThreadLocalRandom] (value [io.netty.util.internal.ThreadLocalRandom@23c5759]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
May 07, 2014 10:42:25 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.util.Recycler$1] (value [io.netty.util.Recycler$1@478a9534]) and a value of type [io.netty.util.Recycler.Stack] (value [io.netty.util.Recycler$Stack@5b49ad9d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
May 07, 2014 10:42:25 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/xxxapps] created a ThreadLocal with key of type [io.netty.channel.ChannelHandlerAdapter$1] (value [io.netty.channel.ChannelHandlerAdapter$1@42254d3d]) and a value of type [java.util.WeakHashMap] (value [{class com.relayrides.pushy.apns.ApnsConnection$1=true}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Okay. Will continue to investigate.
This appears to be an upstream issue. I've opened netty/netty#2522, which includes a minimal test case. Will post here as things develop, but I don't think there's much we can do about it from our end.
The ThreadLocals are created deliberately by Netty, and it is expected that they will remain even after we shut things down. It appears that there is no viable technical solution for this issue from our end. Our recommendation is now to avoid using Pushy in a container environment. I've updated the README in b3b581b.
Hi,
It's been 3 years. I just started using Pushy in my web project and I'm worried about the possible consequences. Is there any news regarding this issue?
Please see https://github.com/relayrides/pushy/wiki/Using-Pushy-in-an-application-container#general-issues. The issues remain, but are not an issue in steady-state operation.
Thanks a lot for your feedback, jchambers. Anyway, I'm concerned by any possible memory leaks or unstable behaviour that this issue might cause. I'll try to figure out a way of tracking these threads and stopping them when instances are closed, anyway.
Anyway, I'm concerned by any possible memory leaks or unstable behaviour that this issue might cause.
Please do as you see fit, but for other readers, I want to be clear that this is not a stability issue, nor does it lead to "leaks" in any conventional sense. Tomcat, by design, clears up any lingering ThreadLocal instances when it shuts down a webapp. It will still issue a warning, but there is no reason to believe there are actual adverse effects.
its closed now though i want to add my opinion on documentation.
this "problem" gets far to much attention in the README. our technical management has tendency to read only headlines and decline to use this library just because they read "memory leaks". they dont read this ticket and "trust" what they read on readme. i had to do hours of explaining why this "problem" is absolutely no reason for any worries to runtime stability.
i dont say hide the warning, but maybe make it less agressive.