Reduce memory usage
Opened this issue · 2 comments
Memory usage of cargo tally can get quite severe. Running cargo tally winapi windows-sys peaked at 12.7 GB, which isn't terrible (but also far from great). However, when trying it with --transitive, the program exhausts my 32 GB of RAM and 8 GB of swap before Linux kills it as OOM, so it apparently uses in excess of 40 GB.
Is it somehow possible to reduce the amount of memory used? I don't mind if it takes longer to run; the speed is reasonable as-is. Perhaps if some data could be written to disk (and later read back as needed) if memory usage exceeds a threshold?
On my machine cargo tally winapi windows-sys --transitive uses 38.2 GB peak, of which the peak amount allocated by Rust is 29.3 GB, and the rest is presumably heap fragmentation or something (I don't know much about this space).
In any case I have a lot more than that, and have never hit OOMs despite using this crate a lot (including concurrent invocations), so I don't plan to look into this, but I would be open to PRs.
That's reasonable. I don't use it often myself, so it's not a huge deal. Given my infrequency of use, I likely won't spend much time (if any) looking at ways to reduce the memory usage.
Feel free to close this unless you'd rather leave it open as "acceptable, but I'm not doing it".