Strippy is fast but fat.
Closed this issue · 4 comments
Strippy needs to lose weight.
I'm watching it process 2.3gb of log files rn and the main process is using a whopping 13gb and climbing. This is a bit of an issue. This is during the scout-stripper routine so there must be something we can do in there to improve either garbage collection or something further as that is a ridiculous amount of memory to consume.
plz fix
I had a thought that we could output the keylist found from each file to a tmp file rather than leaving it in the job system. Piping it to disk is apparently very fast and it could/should slim down what's kept in memory. We could then remove jobs as they're finished.
Clearing the content variable after scouting should also slim strippy down. That might be a fix that can go into Milestone1
Okay, so Strippy was getting fat on error messages and I believe that the key-finding bloatage is now fixed with ebc2342. There was something strange going on with a bad casting error.
Commit 2b3cb4e also prevents the manage-job function from starting/running more than $MaxThreads number of processes at a time. Sometimes the progress of a process would become -1 and that wouldn't register with the collecting program.
5728d14 also fixed a memory issue that was caused by using the older, regex based -replace function rather than the newer (.NET?) .replace function
From using strippy with these changes in place on site I can confirm that Strippy runs much slimmer now. This ticket can be considered resolved