OutOfMemoryError occurs when executed for > 8000 genome

[y-naka]

I want to get CSBs for over 8000 genomes set.
The genomes include only E. coli and have very low phylogenetic diversity.

I excuted following command,

% java -jar /path/to/CSBFinder/CSBFinder-0.5.1-jar-with-dependencies.jar \
>       -in ${INPUT} \
>       -out TXT \
>       -q 1 \
>       -out-dir ${OUTPUT}

and got following errors.

21/10/2019 17:40:00: Parsing input genomes file
21/10/2019 17:41:05: Executing workflow
21/10/2019 17:41:05: Extracting CSBs from 8288 input sequences.
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Integer.valueOf(Integer.java:832)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.multimapAddAll(InstanceNode.java:127)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:115)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113)
        at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCount(InstanceNode.java:98)
        at Model.SuffixTreePatternFinder.SuffixTrees.GeneralizedSuffixTree.computeCount(GeneralizedSuffixTree.java:428)
        at Model.SuffixTreePatternFinder.SuffixTreeAlgorithm.findPatterns(SuffixTreeAlgorithm.java:140)
        at Model.CSBFinderWorkflow.run(CSBFinderWorkflow.java:73)
        at Model.Controller.pipeline(Controller.java:168)
        at Model.Controller.run(Controller.java:65)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

What do you think is the cause of this?

[dinasv]

How much RAM do you have on your computer? If you have 8GB for example, you can run the program using the option "java -Xmx8g" to increase the RAM memory used by the program. In addition, you could try and use the option "-alg match_points" This is a memory efficient algorithm that should decrease the memory requirement. Because you are running the program on a very large datase, I would also advice to change the quorum parameter to "-q 10" or to a larger number, as the program could take a very long time to run. You can also improve the runtime by using the option "-procs 0" to use all available threads on the computer. This option should be combined with the option "-alg match_points"

…

On Mon, Oct 21, 2019, 19:22 y-naka ***@***.***> wrote: I want to get CSBs for over 8000 genomes set. The genomes include only E. coli and have very low phylogenetic diversity. I excuted following command, % java -jar /path/to/CSBFinder/CSBFinder-0.5.1-jar-with-dependencies.jar \> -in ${INPUT} \> -out TXT \> -q 1 \> -out-dir ${OUTPUT} and got following errors. 21/10/2019 17:40:00: Parsing input genomes file 21/10/2019 17:41:05: Executing workflow 21/10/2019 17:41:05: Extracting CSBs from 8288 input sequences. Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Integer.valueOf(Integer.java:832) at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.multimapAddAll(InstanceNode.java:127) at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:115) at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113) at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113) at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCountRecursive(InstanceNode.java:113) at Model.SuffixTreePatternFinder.SuffixTrees.InstanceNode.computeAndCacheCount(InstanceNode.java:98) at Model.SuffixTreePatternFinder.SuffixTrees.GeneralizedSuffixTree.computeCount(GeneralizedSuffixTree.java:428) at Model.SuffixTreePatternFinder.SuffixTreeAlgorithm.findPatterns(SuffixTreeAlgorithm.java:140) at Model.CSBFinderWorkflow.run(CSBFinderWorkflow.java:73) at Model.Controller.pipeline(Controller.java:168) at Model.Controller.run(Controller.java:65) Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main" What do you think is the cause of this? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=ACWSNFFLO4YZOGUDX6OMR4LQPV7E5A5CNFSM4JC3H272YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HTEAFAA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWSNFCFMZIAG7LJBM57I6LQPV7E5ANCNFSM4JC3H27Q> .

[y-naka]

I tried the option java -Xmx10g and everything went well with us.

Thank you for nice advise.