Guppy v6.5.7 with pod5 uses tons of memory vs fast5

Question

Guppy v6.5.7 with pod5 uses tons of memory vs fast5

Closed this issue a year ago · 3 comments

billytcl commented a year ago

I converted fast5s to a ~900 GB single pod5 file (v0.2). When basecalling, it is taking up a ton of memory:

This doesn't happen when I call from fast5s. Is this normal?

Answer 1 · 2023-06-12T08:01:09.000Z

Hi @billytcl,

This is due to how pod5 uses virtual memory, it's to do with how data is cached in case its needed again.

Does the basecall run ignoring the high memory use?

We will look at changing config to avoid this memory usage.

Thanks,

George

Answer 2 · 2023-06-12T08:02:19.000Z

As long as it won't crash the system then it's all good!

…

On Mon, Jun 12, 2023 at 1:01 AM jorj1988 ***@***.***> wrote: Hi @billytcl <https://github.com/billytcl>, This is due to how pod5 uses virtual memory, it's to do with how data is cached in case its needed again. Does the basecall run ignoring the high memory use? We will look at changing config to avoid this memory usage. Thanks, - George — Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPHYT24NG3JOVQBP2KGGZDXK3EFBANCNFSM6AAAAAAZC4OMK4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Answer 3 · 2023-06-12T08:08:14.000Z

Hi @billytcl,

It's OS managed, so will be released once another process needs the memory. It should be harmless, but I agree it can look worrying.

George