Started gap closing procedure Child process crashed
dbajpp0 opened this issue · 3 comments
I ran the fasta reads of two HiFi cells (24 and 22 Gb) in LJA in a single AMD EPYC 7351P 16-Core Processor but I got a crash at gap closing step (about 30 hours of running). RAM Memory is about 128 Gb and the maximum used was about 100Gb, while crashed it was using about 55 Gb. The DNA to make both libraries is from a pool of 20 tiny individuals (> 1 mm each). hifiasm finished the analyses. I build the binary 5 days ago from git in an ubuntu Ubuntu 20.04.5, kernel 5.4.0, gcc 9.4.0 and cmake 3.16.3.
Any tip to avoid crash and restart a new run from crashing point?
Thanks in advance,
Joan
log output
lja -o tethys_lajolla_2runs --reads m64086e_230106_194450.hifi_reads.fastq.gz --reads m64086e_230111_202349.hifi_reads.fastq.gz -t 28 --diploid
00:00:00 4Mb INFO: Hello! You are running La Jolla Assembler (LJA), a tool for genome assembly from PacBio HiFi reads
00:00:00 5Mb INFO: LJA pipeline started
00:00:00 5Mb INFO: Performing initial correction with k = 501
00:00:00 0Mb INFO: Reading reads
00:00:00 0Mb INFO: Extracting minimizers
00:14:18 28.8Gb INFO: Finished read processing
00:14:18 28.8Gb INFO: 36073726 hashs collected. Starting sorting.
00:14:20 29.9Gb INFO: Finished sorting. Total distinct minimizers: 9104109
00:14:20 29.9Gb INFO: Starting construction of sparse de Bruijn graph
00:14:32 32.1Gb INFO: Vertex map constructed.
00:14:32 32.1Gb INFO: Filling edge sequences.
00:40:24 47.4Gb INFO: Finished sparse de Bruijn graph construction.
00:40:26 47.4Gb INFO: Collecting tips
00:40:31 49.7Gb INFO: Added 2420389 artificial minimizers from tips.
00:40:31 49.7Gb INFO: Collected 30484083 old edges.
00:40:34 49.7Gb INFO: New minimizers added to sparse graph.
00:40:34 49.7Gb INFO: Refilling graph with old edges.
00:55:41 49.7Gb INFO: Filling graph with new edges.
00:57:01 49.7Gb INFO: Finished fixing sparse de Bruijn graph.
00:57:56 50.9Gb INFO: Starting to extract disjointigs.
00:58:41 50.9Gb INFO: Finished extracting 13298115 disjointigs of total size 16243978545
01:03:36 0Mb INFO: Loading disjointigs from file "tethys_lajolla_2runs/k501/disjointigs.fasta"
01:08:34 42.8Gb INFO: Filling bloom filter with k+1-mers.
01:41:44 42.9Gb INFO: Filled 40377766360 bits out of 306611933760
01:41:44 42.9Gb INFO: Finished filling bloom filter. Selecting junctions.
02:08:39 46.7Gb INFO: Collected 24431404 junctions.
02:10:36 46.7Gb INFO: Starting DBG construction.
02:11:00 46.7Gb INFO: Vertices created.
02:18:25 46.7Gb INFO: Filled dbg edges. Adding hanging vertices
02:18:33 46.7Gb INFO: Added 274 hanging vertices
02:18:33 46.7Gb INFO: Merging unbranching paths
02:18:59 46.7Gb INFO: Ended merging edges. Resulting size 22382185
02:22:14 46.7Gb INFO: Cleaning edge coverages
02:22:26 46.7Gb INFO: Collecting alignments of sequences to the graph
02:36:24 63.1Gb INFO: Alignment collection finished. Total length of alignments is 578668637
02:37:15 63.1Gb INFO: Precorrecting reads
02:38:00 63.1Gb INFO: Applying corrections to reads
02:38:21 63.1Gb INFO: Applied correction to 1404153 reads
02:38:21 63.1Gb INFO: Corrected simple errors in 1404153 reads
02:38:21 63.1Gb INFO: Applying changes to the graph
03:13:12 89.5Gb INFO: Collecting and storing read suffixes
03:43:45 89.5Gb INFO: Correcting dinucleotide errors in reads
03:53:05 89.5Gb INFO: Applying corrections to reads
03:58:42 89.5Gb INFO: Applied correction to 1112277 reads
03:58:42 89.5Gb INFO: Corrected 1112277 dinucleotide sequences
03:58:42 89.5Gb INFO: Marking reliable edges
03:59:21 89.5Gb INFO: Marked 3271893 edges in 915644 paths as reliable
03:59:23 89.5Gb INFO: Correcting low covered regions in reads with K = 800
04:31:25 89.5Gb INFO: Applying corrections to reads
04:55:52 89.5Gb INFO: Applied correction to 2332968 reads
04:55:52 89.5Gb INFO: Corrected low covered regions in 2332968 reads with K = 800
04:55:52 89.5Gb INFO: Applying changes to the graph
07:57:06 94.7Gb INFO: Marking reliable edges
07:57:26 94.7Gb INFO: Marked 996751 edges in 462739 paths as reliable
07:57:26 94.7Gb INFO: Correcting low covered regions in reads with K = 2000
09:28:24 94.7Gb INFO: Applying corrections to reads
09:41:48 94.7Gb INFO: Applied correction to 358442 reads
09:41:48 94.7Gb INFO: Corrected low covered regions in 358442 reads with K = 2000
09:41:48 94.7Gb INFO: Applying changes to the graph
13:19:40 96.7Gb INFO: Correcting dinucleotide errors in reads
13:38:37 96.7Gb INFO: Applying corrections to reads
13:40:21 96.7Gb INFO: Applied correction to 33011 reads
13:40:21 96.7Gb INFO: Corrected 33011 dinucleotide sequences
13:40:21 96.7Gb INFO: Marking reliable edges
13:40:41 96.7Gb INFO: Marked 864704 edges in 421537 paths as reliable
13:40:42 96.7Gb INFO: Correcting low covered regions in reads
18:54:15 101.8Gb INFO: Applying corrections to reads
20:54:46 101.8Gb INFO: Applied correction to 1103409 reads
20:54:48 101.8Gb INFO: Corrected low covered regions in 1516510 reads
20:54:49 101.8Gb INFO: Marking reliable edges
20:55:22 101.8Gb INFO: Marked 715265 edges in 339678 paths as reliable
20:55:22 101.8Gb INFO: Correcting low covered regions in reads with K = 3500
22:33:29 101.8Gb INFO: Applying corrections to reads
22:41:14 101.8Gb INFO: Applied correction to 95477 reads
22:41:14 101.8Gb INFO: Corrected low covered regions in 95477 reads with K = 3500
22:41:14 101.8Gb INFO: Applying changes to the graph
26:02:43 101.8Gb INFO: Printing reads to fasta file "tethys_lajolla_2runs/k501/corrected.fasta"
26:22:36 5Mb INFO: Initial correction results with k = 501 printed to "tethys_lajolla_2runs/k501/corrected.fasta"
26:22:36 5Mb INFO: Performing second phase of error correction using k = 5001
26:22:36 0Mb INFO: Reading reads
26:22:36 0Mb INFO: Extracting minimizers
26:28:02 20.3Gb INFO: Finished read processing
26:28:02 20.3Gb INFO: 71226089 hashs collected. Starting sorting.
26:28:05 22.3Gb INFO: Finished sorting. Total distinct minimizers: 27526106
26:28:05 22.3Gb INFO: Starting construction of sparse de Bruijn graph
26:28:51 32.7Gb INFO: Vertex map constructed.
26:28:51 32.7Gb INFO: Filling edge sequences.
26:42:05 78.9Gb INFO: Finished sparse de Bruijn graph construction.
26:42:14 78.9Gb INFO: Collecting tips
26:42:27 82Gb INFO: Added 1815686 artificial minimizers from tips.
26:42:27 82Gb INFO: Collected 54232201 old edges.
26:42:31 82.4Gb INFO: New minimizers added to sparse graph.
26:42:31 82.4Gb INFO: Refilling graph with old edges.
26:55:32 82.4Gb INFO: Filling graph with new edges.
26:55:59 83.2Gb INFO: Finished fixing sparse de Bruijn graph.
26:58:44 86.3Gb INFO: Starting to extract disjointigs.
26:59:41 86.3Gb INFO: Finished extracting 2351791 disjointigs of total size 18852733867
27:04:43 0Mb INFO: Loading disjointigs from file "tethys_lajolla_2runs/k5001/disjointigs.fasta"
27:10:10 32.5Gb INFO: Filling bloom filter with k+1-mers.
27:34:14 32.5Gb INFO: Filled 32816977551 bits out of 226925666432
27:34:14 32.5Gb INFO: Finished filling bloom filter. Selecting junctions.
27:54:01 33Gb INFO: Collected 5459018 junctions.
27:54:30 33Gb INFO: Starting DBG construction.
27:54:35 33Gb INFO: Vertices created.
27:59:21 33Gb INFO: Filled dbg edges. Adding hanging vertices
27:59:23 33Gb INFO: Added 450 hanging vertices
27:59:23 33Gb INFO: Merging unbranching paths
27:59:37 33Gb INFO: Ended merging edges. Resulting size 2770710
28:01:49 33Gb INFO: Cleaning edge coverages
28:01:50 33Gb INFO: Collecting alignments of sequences to the graph
28:01:50 33Gb INFO: Storing suffixes of read paths of length up to 10000000
28:06:52 44.3Gb INFO: Alignment collection finished. Total length of alignments is 10881215
28:06:53 44.3Gb INFO: Correcting dinucleotide errors in reads
28:06:55 44.3Gb INFO: Applying corrections to reads
28:06:55 44.3Gb INFO: Applied correction to 134 reads
28:06:55 44.3Gb INFO: Corrected 134 dinucleotide sequences
28:06:55 44.3Gb INFO: Marking reliable edges
28:06:59 44.3Gb INFO: Marked 410100 edges in 114968 paths as reliable
28:06:59 44.3Gb INFO: Correcting low covered regions in reads
28:34:32 48.5Gb INFO: Applying corrections to reads
28:34:34 48.5Gb INFO: Applied correction to 507122 reads
28:34:34 48.5Gb INFO: Corrected low covered regions in 507608 reads
28:34:34 48.5Gb INFO: Collapsing bulges
28:34:35 48.5Gb INFO: Applying corrections to reads
28:34:35 48.5Gb INFO: Applied correction to 8052 reads
28:34:35 48.5Gb INFO: Collapsed bulges in 17964 reads
28:34:35 48.5Gb INFO: Applying changes to the graph
28:55:11 55.7Gb INFO: Running second round of error correction
28:55:11 55.7Gb INFO: Correcting dinucleotide errors in reads
28:55:12 55.7Gb INFO: Applying corrections to reads
28:55:12 55.7Gb INFO: Applied correction to 19 reads
28:55:12 55.7Gb INFO: Corrected 19 dinucleotide sequences
28:55:12 55.7Gb INFO: Correcting dinucleotide errors in reads
28:55:14 55.7Gb INFO: Applying corrections to reads
28:55:14 55.7Gb INFO: Applied correction to 2 reads
28:55:14 55.7Gb INFO: Corrected 2 dinucleotide sequences
28:55:14 55.7Gb INFO: Marking reliable edges
28:55:16 55.7Gb INFO: Marked 91028 edges in 43137 paths as reliable
28:55:16 55.7Gb INFO: Correcting low covered regions in reads
29:02:42 55.7Gb INFO: Applying corrections to reads
29:02:43 55.7Gb INFO: Applied correction to 73052 reads
29:02:43 55.7Gb INFO: Corrected low covered regions in 73372 reads
29:02:43 55.7Gb INFO: Correcting dinucleotide errors in reads
29:02:45 55.7Gb INFO: Applying corrections to reads
29:02:45 55.7Gb INFO: Applied correction to 2 reads
29:02:45 55.7Gb INFO: Corrected 2 dinucleotide sequences
29:02:45 55.7Gb INFO: Remarking reliable edges
29:02:52 55.7Gb INFO: Correcting tips using reliable edge marks
29:11:15 55.7Gb INFO: Applying corrections to reads
29:11:15 55.7Gb INFO: Applied correction to 316740 reads
29:11:15 55.7Gb INFO: Collapsing bulges
29:11:16 55.7Gb INFO: Applying corrections to reads
29:11:16 55.7Gb INFO: Applied correction to 549 reads
29:11:16 55.7Gb INFO: Collapsed bulges in 16192 reads
29:11:16 55.7Gb INFO: Applying changes to the graph
29:28:21 55.7Gb INFO: Started gap closing procedure
Child process crashed
The last three lines of the log file are:
29:28:21 55.7Gb INFO: Started gap closing procedure
29:28:22 55.7Gb TRACE: Collecting k-mers from tips
29:28:44 96.7Gb TRACE: Sorting k-mers from tips
I have encountered the same problem. Have you resolved it ?
Dear developer, I have also encontered this isuue:
24:31:58 73.3Gb INFO: Finished alignment.
24:31:58 73.3Gb INFO: Printing alignments to "/data/02_way03_lja/01_assembly/output/uncompressing/alignments.txt"
24:32:47 73.8Gb INFO: Reading and processing initial reads from ["P3.hifireads.fasta"]
26:19:39 75.2Gb INFO: Uncompressing homopolymers in contigs
Child process crashed