High memory usage because of shard reaper createReadStream
Closed this issue · 2 comments
Package Versions
Replace the values below using the output from npm list storj. Use npm list -g storj if installed globally.
Storj Share 4.0.11
Core Library 6.2.0
Protocol 1.1.0
Electron 1.6.1
Chrome 56.0.2924.87
Replace the values below using the output from node --version.
v6.10.0
Expected Behavior
Please describe the program's expected behavior. Include an example of your
usage code in the back ticks below if applicable.
My GUI is running at ~100MB and stable. I would expect that the shard reaper needs a few MB more.
Actual Behavior
While the shard reaper is running my GUI needs more than 1GB and I have only 200GB used space. Sooner or later I should get an out of memory exception.
GUI memory snapshot show me an Array with 25246 Strings referenced by: https://github.com/Storj/core/blob/af03c532a66e1e16904d2fe38511084b87a5878e/lib/storage/adapter.js#L177
Steps to Reproduce
Please include the steps the reproduce the issue, numbered below. Include as
much detail as possible.
- Run GUI with reduced reaper interval
- Watch memory usage
Ooh yeah, we are buffering all the shard keys before doing the scan. That's a problem. Will rework that.
A few hours later the cleanup job was finished. IO in idle mode. Memory was still 100MB higher than expected but I was not able to find that memory leak. I will wait for your fix, try again and open a new issue if needed.