master branch memory leak
Closed this issue · 2 comments
ChillFish8 commented
Running heavy tests with large datasets master currently shows signs of a memory leak when running the 50 million amazon dataset.
0.7.1 Does not have this issue and successfully runs at ~9GB memory usage max when indexing the dataset and 4.8GB when complete.
The total data is ~26GB but ram usage on master crept up to 62GB before being killed by OOM.
ChillFish8 commented
Doing some debugging this is likely a consequence of having a new frequency set for each document
ChillFish8 commented
Fixed in #61