Krackle with tomcat
Opened this issue · 9 comments
I am using krackle-0.8.1.2 in my web application which is hosted on tomcat 7. I can still find a PID after the tomcat shutdown, and it kept printing "WARN c.b.bdp.krackle.producer.Producer - Trying to send data on a closed producer." in log file until the process was killed. producer.close() function was called when the application was going down when I brought the tomcat down. And this is same for krackle-0.7.7 and krackle-0.8.0. Do you have any idea about this and show some advice about this? Thanks.
Hey @DeanHere I'm not very familiar with Tomcat, however can you confirm that there were no attempts at sending a message after the producer was closed? If you can describe how you are using Krackle with Tomcat then I might be able to dig in a little deeper...
Hey @ariens thanks for reply. I just created a producer instance, did nothing, no "producer.send()" was called, and close it when tomcat was brought down.
I just sent my test code to your email dave@ariens.ca, there is a log string as below in the file of 'catalina.out' which is the default tomcat log file. This is showing there still resources cannot be released after the tomcat was brought down. And I am not able to reproduce the "WARN c.b.bdp.krackle.producer.Producer - Trying to send data on a closed producer." log in this test code, I would try another way.
SEVERE: The web application [/KrackleProducer] appears to have started a thread named [metrics-console-reporter-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
CentOS 6.6 x64, Tomcat 7.0.62, kafka_2.9.2-0.8.1.1.tgz
@DeanHere, thanks for following up with your code and Tomcat config. I'm currently experiencing some downtime with my ariens.ca email account, feel free to send the logs to dariens@ NO SPAM blackberry.com and I'll review over the weekend. Thanks.
@ariens Just would like to know that if there is a workaround for this, to start / stop the threads manually by some public methods. I had fwed the email to dariens@ blackberry.com last week and hope you had got the test code. thanks.
Hey @DeanHere, I haven't had a chance to attempt to replicate. I suspect that you're dealing with a nuance of Tomcat, though. Since the introduction of Krackke to the BlackBerry infrastructure we've processed billions of logs a day and never ran into this problem. Since there are no process related long lived threads to Krackke and I am not experienced with Tomcat I can only infer that you are somehow calling send once a closed producer. If you remove Tomcat from the mix does this error message go away? From: DeanHereSent: Thursday, August 6, 2015 5:22 AMTo: blackberry/KrackleReply To: blackberry/KrackleCc: Dave AriensSubject: Re: [Krackle] Krackle with tomcat (#6)@ariens Just would like to know that if there is a workaround for this, I had fw the email to dariens@ blackberry.com last week and hope you had got the test code. thanks.
—Reply to this email directly or view it on GitHub.
"SEVERE: The web application [/KrackleProducer] appears to have started a thread named [metrics-console-reporter-1-thread-1] but has failed to stop it. This is very likely to create a memory leak."
@ariens The scheduledExecutor which is not shutdown when the close() method was called in the Producer class, this is the root cause of my issue I guess.
One more thing, as I mentioned before, above message was in my log file. The metrics-console-reporter is from metrics-core-3.1.1.jar which is calling by com.blackberry.bdp.common.jmx.MetricRegistrySingleton in your BB-BigData-Common-Utils-0.0.9 project when a Producer is created, but was it closed when a Producer is closing? If it is not, this is very likely to create a memory leak when using tomcat.
All unused threads should be closed for sure aren't they?
Would you please confirm this? Thanks a lot.
@DeanHere, awesome find. I will check in the console reporter (not frequently enabled in our stack) and patch if that's the culprit... From: DeanHereSent: Friday, August 7, 2015 6:46 AMTo: blackberry/KrackleReply To: blackberry/KrackleCc: Dave AriensSubject: Re: [Krackle] Krackle with tomcat (#6)"SEVERE: The web application [/KrackleProducer] appears to have started a thread named [metrics-console-reporter-1-thread-1] but has failed to stop it. This is very likely to create a memory leak."
@ariens The scheduledExecutor which is not shutdown when the close() method was called in the Producer class, this is the root cause of my issue I guess.
One more thing, as I mentioned before, above message was in my log file. The metrics-console-reporter is from metrics-core-3.1.1.jar which is calling by com.blackberry.bdp.common.jmx.MetricRegistrySingleton in your BB-BigData-Common-Utils-0.0.9 project when a Producer is created, but was it closed when a Producer is closing? If it is not, this is very likely to create a memory leak when using tomcat.
All unused threads should be closed for sure aren't they?
Would you please confirm this? Thanks a lot.
—Reply to this email directly or view it on GitHub.
@ariens Thanks. Just in case if you missed the root cause of my issue, the variable of "scheduledExecutor" in com.blackberry.bdp.krackle.producer, which is not shutdown when the close() method was called.
And about the console reporter, it is enable by default and it seems like there is no option to disable it in this version of krackle.