Possible memory leaks
Opened this issue · 2 comments
Apologies in advance that this issue is not very concrete! I'll elaborate more once I get a chance to do some proper profiling.
After running this benchmark on the forward chaining reasoning components in Comunica I've been noticing that my laptop appears to have to deal with a large number of memory leaks when it shuts down. I suspect it is either something wrong internally with the AsyncIterator, or the way that I am using it in that case.
my laptop appears to have to deal with a large number of memory leaks when it shuts down
Interesting. Does that imply that there's memory that isn't being returned to the OS even after the node process has gone?
For non-trivial memory leaks I suggest testing with jemalloc for a more complete picture of what is going on. It's relatively easy to do so and it's well documented at https://sharp.pixelplumbing.com/install#linux-memory-allocator .
I've already encountered three different cases in entirely different contexts in which RSS memory usage would grow indefinitely, diverging from stable heap memory usage. In one use case, memory appeared to never be returned back to the OS even upon service restarts. Furthermore, this behavior was not consistent across platforms (macOS x86_64 would show the issue and so would linux x86_64, whereas macOS arm64 would be fine).
Perusing through the issues at https://github.com/lovell/sharp/issues?q=is%3Aissue+jemalloc got me on the right track.
Tagging @rubensworks as this might be relevant for Comunica, too.
Similarly I think we should try testing with this too https://facebookincubator.github.io/memlab/docs/intro.