NCBI-Hackathons/NovoGraph

CRAM2VCF.cpp needs a lot of memory

TorHou opened this issue · 1 comments

TorHou commented

When running launch_CRAM2VCF_C++.pl I have noticed that running 10 processes in parallel while(scalar(keys %still_running) >= 10)will use up a lot of mermory. Getting a job killed with 1024 Gbyte of RAM.

So I had to change it to while(scalar(keys %still_running) >= 2) in code.

  • IMHO it would be a good to make this an adjustable parameter.
  • it might be a good idea to look at why CRAM2VCF.cpp needs so much memory and if this can be changed
evanbiederstedt commented

Hi @TorHou

When running launch_CRAM2VCF_C++.pl I have noticed that running 10 processes in parallel while(scalar(keys %still_running) >= 10)will use up a lot of mermory. Getting a job killed with 1024 Gbyte of RAM.

So I had to change it to while(scalar(keys %still_running) >= 2) in code.

Yes, you're referring to lines: https://github.com/NCBI-Hackathons/NovoGraph/blob/master/scripts/launch_CRAM2VCF_C%2B%2B.pl#L102-L116

There's certainly no reason why this couldn't be a parameter in the script.

it might be a good idea to look at why CRAM2VCF.cpp needs so much memory and if this can be changed

This isn't too surprising, if I recall. I agree though---the question to my mind is how easily things could be improved in terms of , and if it's worth the investment. (Not the most helpful comment I know, but I'll have to take a closer look at the *cpp code).