performancecopilot/parfait

Parfait appears to leak TimerTasks

jerdogma opened this issue · 2 comments

On linux, parfait appears to leak TimerTask objects, which in turn leak mmap files.

We are using parfait in our JVM as an agent. It starts and runs normally, we can add instances to it as we need to.

We observe that when we add an instance, the JVM seems to hold onto old versions of the memory mapped file:

root@host:~# grep /var/lib/pcp/tmp/mmv/com /proc/$(cat /var/run/PID)/maps
7f2754067000-7f275406d000 rw-s 00000000 00:14 50972 /var/lib/pcp/tmp/mmv/com
7f2764002000-7f2764007000 rw-s 00000000 00:14 50292 /var/lib/pcp/tmp/mmv/com (deleted)
7f276402e000-7f2764032000 rw-s 00000000 00:14 49529 /var/lib/pcp/tmp/mmv/com (deleted)

We also see that this increases over time, as we add new instances.

We took heap dumps before and after noticing this behavior, and it appears that:

  • TimerTask is an anonymous function, which QuiescentRegistryListeners create as necessary
  • Each TimerTask holds a DynamicMonitoringViews
  • Each DynamicMonitoringViews holds a DirectByteBuffer
  • Each DirectByteBuffer corresponds to an entry in the maps file

When we changed QuiescentRegistryListener to:

  • Track the previously created TimerTask
  • When it makes a new TimerTask, cancel the old one
    Then, the JVM would GC the memory mapped files.

I'm attaching the patch we created. I offer this contribution subject to the apache 2.0 license that governs parfait itself.

Thanks for all the work you and your team do on parfait, we really like PCP, it does a great job for us.

leak-minimal.txt

This PR should fix the QuiescentRegistryListener leak: #122