Bigsi for 600000 genomes
Closed this issue · 4 comments
Hello,
I work in a ministry of health in the pathogenes detection.
I read about Bigsi (congratulation!), and I think it could be useful.
However didn't find the docs very useful (sorry), and the demo link https://bigsi.readme.io/ doesn't work
We need to find presence of specific genes in 600.000 Salmonella genomes.
I am in charge to find the better tool to do that.
Could you tell me please if you think it's doable with Bigsi, and how many volume in my hard drive I need,
and how much time (approximatively of course) it would take?
Thank you very much
David
Hello,
I would recommend you to use COBS (https://github.com/bingmann/cobs) instead of BIGSI. It is a "reimplementation" of BIGSI, but uses compacted indexes (i.e. less disk space is required) and is much faster. However, it relies on having random access to the disk (e.g. SSD, flash filesystems, etc), so it wouldn't work well on hard disk filesystems. You also have the option of loading the whole index into memory, which would be heavy on RAM usage, but would work on any filesystem. I recently made a similar query to yours. I queried 367 sequences against a database of 661k bacterial genomes, and that took 22 hours and 3 GBs of RAM. But I should warn you that I think this runtime is slow because COBS is constantly making disk access (I did not tell it to load the entire index into RAM as it is 900GB), and although the disk is SSD, it is a shared cluster with hundreds of users, so accessing the disk is known to be slow. If you have a random-access disk for yourself, or with few users, it should be way faster
Thank you for reaching out
So I'll learn about COBS
I'm new in the bioinformatics field, so before to do task so big like this, I prefer to ask the community (I don't want to burn my boss's computer :) )
I'll check with the IT team what it's the cleverer thing to do regarding the compute area
Thank you so much for your help
No problems! You could subsample your dataset to 1k genomes and try building and querying a COBS index to familiarise yourself with the tool.
Cheers
Thank you for the tips!