Memory management with m5nr-tools.pl

Question

Memory management with m5nr-tools.pl

Closed this issue 11 years ago · 6 comments

I was running m5nr-tools.pl to get annotations from protein ids and the script quickly consumed all 8GB of memory on my machine and would not release it after shutdown. I tried increasing the batch size to 1000 which helped a bit but did not solve the problem.

The command I ran was:
perl m5nr-tools.pl --api http://api.metagenomics.anl.gov --option annotation --source RefSeq --acc ../acc.txt > ../all-annotation2.txt

The query file is 386M and contains 28,581,905 protein ids.

Answer 1 · 2014-03-10T14:55:33.000Z

Thanks for letting us know. We will check. Is there a way you could provide us with your query file?

Answer 2 · 2014-03-10T18:37:39.000Z

Sure. the file is here http://tinyecology.com/GIS/acc.txt.zip

Answer 3 · 2014-03-10T23:37:23.000Z

Thanks for the file, I tried to reproduce the behavior but did not get the same results. Anyhow memory usage went up to 4GB and slowly went down over the whole runtime. The memory usage should be improved.

Answer 4 · 2014-03-11T00:04:20.000Z

This might be an issue of the perl version being used and/or the OS.
On Mar 10, 2014, at 6:37 PM, Andreas Wilke notifications@github.com wrote:

Thanks for the file, I tried to reproduce the behavior but did not get the same results. Anyhow memory usage went up to 4GB and slowly went down over the whole runtime. The memory usage should be improved.

—
Reply to this email directly or view it on GitHub.

Answer 5 · 2014-03-11T18:06:54.000Z

I'm using Perl 5.14.2 on an Ubuntu machine:
This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi
3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Reducing the size of the file solved the problem.

Answer 6 · 2014-05-06T20:38:54.000Z

Updated script - has reduced memory footprint. Memory usage still depends on input and output size.