Canu v2.3 Failed with 'Segmentation fault'; backtrace (libbacktrace)
Lixx1989 opened this issue · 6 comments
Hi,
I repeatedly encountered this error and I don't seem to be able to pinpoint the problem.
I'm running canu on a linux grid, with 1 node, 128 cores, and enough storage space left. My assembly is about 1.5g.
--Command--
/media/disk1/software/canu/build/bin/canu -p bx_canu -d assembly genomeSize=1.5g useGrid=false --pacbio-hifi /media/disk1/xxli/hifi/CRR783774.fq.gz
--Version--
canu snapshot v2.3-development +139 changes (r10410 a55ecfa)
About four consensus jobs failed. I am pasting the error message from of 3 consensus files (consensus.000010.out and ). And they are different error messages.
**consensus.0000010.out:**
Found perl:
/usr/local/Miniconda3/bin/perl
This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi
Found java:
/usr/local/Miniconda3/bin/java
openjdk version "11.0.13" 2021-10-19
Found canu:
/media/disk1/software/canu/build/bin/canu
canu snapshot v2.3-development +139 changes (r10410 a55ecfaa3fa1d39ba7d0a577dcc5e4cd6c413ca0)
Running job 10 based on command line options.
-- Using seqFile '../bx_canu.ctgStore/partition.0010'.
-- Opening tigStore '../bx_canu.ctgStore' version 1.
-- Opening output results file './ctgcns/0010.cns.WORKING'.
--
-- Computing consensus for b=1 to e=483092 with errorRate 0.0500 (max 0.0500) and minimum overlap 500
--
Loading corrected-trimmed reads from seqFile '../bx_canu.ctgStore/partition.0010'
alignEdLib()-- WARNING: tigbgn 15777533 > tigend 15777453 - tiglen 15777453 utgpos 15777783-15792266 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15792266 - tiglen 15777453 utgpos 15777783-15792266 padding 250
alignEdLib()-- WARNING: tigbgn 15778098 > tigend 15777453 - tiglen 15777453 utgpos 15778348-15792155 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15792155 - tiglen 15777453 utgpos 15778348-15792155 padding 250
alignEdLib()-- WARNING: tigbgn 15780215 > tigend 15777453 - tiglen 15777453 utgpos 15780465-15797676 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15797676 - tiglen 15777453 utgpos 15780465-15797676 padding 250
alignEdLib()-- WARNING: tigbgn 15787821 > tigend 15777453 - tiglen 15777453 utgpos 15788071-15802555 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15802555 - tiglen 15777453 utgpos 15788071-15802555 padding 250
alignEdLib()-- WARNING: tigbgn 15789143 > tigend 15777453 - tiglen 15777453 utgpos 15789393-15808908 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15808908 - tiglen 15777453 utgpos 15789393-15808908 padding 250
alignEdLib()-- WARNING: tigbgn 15794263 > tigend 15777453 - tiglen 15777453 utgpos 15794513-15812418 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15812418 - tiglen 15777453 utgpos 15794513-15812418 padding 250
alignEdLib()-- WARNING: tigbgn 15794638 > tigend 15777453 - tiglen 15777453 utgpos 15794888-15811150 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15811150 - tiglen 15777453 utgpos 15794888-15811150 padding 250
alignEdLib()-- WARNING: tigbgn 15794690 > tigend 15777453 - tiglen 15777453 utgpos 15794940-15810271 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 15810271 - tiglen 15777453 utgpos 15794940-15810271 padding 250
Failed with '
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
Failed with 'Segmentation fault'; backtrace (libbacktrace):
**consensus.000011.out**
Found perl:
/usr/local/Miniconda3/bin/perl
This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi
Found java:
/usr/local/Miniconda3/bin/java
openjdk version "11.0.13" 2021-10-19
Found canu:
/media/disk1/software/canu/build/bin/canu
canu snapshot v2.3-development +139 changes (r10410 a55ecfaa3fa1d39ba7d0a577dcc5e4cd6c413ca0)
Running job 11 based on command line options.
-- Using seqFile '../bx_canu.ctgStore/partition.0011'.
-- Opening tigStore '../bx_canu.ctgStore' version 1.
-- Opening output results file './ctgcns/0011.cns.WORKING'.
--
-- Computing consensus for b=1 to e=483092 with errorRate 0.0500 (max 0.0500) and minimum overlap 500
--
Loading corrected-trimmed reads from seqFile '../bx_canu.ctgStore/partition.0011'
alignEdLib()-- WARNING: tigbgn 12561794 > tigend 12561529 - tiglen 12561529 utgpos 12562044-12577933 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12577933 - tiglen 12561529 utgpos 12562044-12577933 padding 250
alignEdLib()-- WARNING: tigbgn 12562649 > tigend 12561529 - tiglen 12561529 utgpos 12562899-12581268 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12581268 - tiglen 12561529 utgpos 12562899-12581268 padding 250
alignEdLib()-- WARNING: tigbgn 12562933 > tigend 12561529 - tiglen 12561529 utgpos 12563183-12579655 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12579655 - tiglen 12561529 utgpos 12563183-12579655 padding 250
alignEdLib()-- WARNING: tigbgn 12563132 > tigend 12561529 - tiglen 12561529 utgpos 12563382-12581868 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12581868 - tiglen 12561529 utgpos 12563382-12581868 padding 250
alignEdLib()-- WARNING: tigbgn 12563648 > tigend 12561529 - tiglen 12561529 utgpos 12563898-12577145 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12577145 - tiglen 12561529 utgpos 12563898-12577145 padding 250
alignEdLib()-- WARNING: tigbgn 12563669 > tigend 12561529 - tiglen 12561529 utgpos 12563919-12579944 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12579944 - tiglen 12561529 utgpos 12563919-12579944 padding 250
alignEdLib()-- WARNING: tigbgn 12565106 > tigend 12561529 - tiglen 12561529 utgpos 12565356-12580020 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12580020 - tiglen 12561529 utgpos 12565356-12580020 padding 250
alignEdLib()-- WARNING: tigbgn 12565524 > tigend 12561529 - tiglen 12561529 utgpos 12565774-12583429 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 12583429 - tiglen 12561529 utgpos 12565774-12583429 padding 250
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
Failed with 'Segmentation fault'; backtrace (libbacktrace):
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utility/src/align/edlib.C::1600 in transformSequences()
utility/src/align/edlib.C::200 in _ZN9merylutil5align5edlib2v110edlibAlignEPKciS4_iNS2_16EdlibAlignConfigE()
utgcns/unitigConsensus.C::750 in _Z10alignEdLibR12dagAlignmentR10tgPositionPcjS3_jdb()
utgcns/unitigConsensus.C::933 in _ZN15unitigConsensus13generatePBDAGEP5tgTigcRSt3mapIjP6sqReadSt4lessIjESaISt4pairIKjS4_EEE._omp_fn.0()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
Segmentation fault (core dumped)
**consensus.000017.out**
Found perl:
/usr/local/Miniconda3/bin/perl
This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi
Found java:
/usr/local/Miniconda3/bin/java
openjdk version "11.0.13" 2021-10-19
Found canu:
/media/disk1/software/canu/build/bin/canu
canu snapshot v2.3-development +139 changes (r10410 a55ecfaa3fa1d39ba7d0a577dcc5e4cd6c413ca0)
Running job 17 based on command line options.
-- Using seqFile '../bx_canu.ctgStore/partition.0017'.
-- Opening tigStore '../bx_canu.ctgStore' version 1.
-- Opening output results file './ctgcns/0017.cns.WORKING'.
--
-- Computing consensus for b=1 to e=483092 with errorRate 0.0500 (max 0.0500) and minimum overlap 500
--
Loading corrected-trimmed reads from seqFile '../bx_canu.ctgStore/partition.0017'
----------CONTAINED READS---------- -DOVETAIL READS-
tigID length reads used coverage ignored coverage used coverage
------- --------- ------- -------- -------- -------- -------- -------- --------
2 22455 11 7 3.24x 0 0.00x 3 2.04x
3 20784 7 3 1.60x 0 0.00x 3 1.96x
4 19497 6 2 1.04x 0 0.00x 3 1.68x
5 24595 4 0 0.00x 0 0.00x 3 1.52x
6 25625 3 1 0.48x 0 0.00x 1 0.57x
8 22405 3 0 0.00x 0 0.00x 2 1.27x
9 27555 5 0 0.00x 0 0.00x 2 0.89x
10 26363 9 1 0.46x 0 0.00x 3 1.45x
11 14437 3 0 0.00x 0 0.00x 2 1.66x
12 21846 5 1 0.46x 0 0.00x 1 0.74x
13 17435 4 0 0.00x 0 0.00x 2 1.17x
14 17587 4 0 0.00x 0 0.00x 2 1.58x
16 16163 3 0 0.00x 0 0.00x 2 1.36x
17 26672 5 0 0.00x 0 0.00x 2 1.01x
18 26056 4 0 0.00x 0 0.00x 2 1.03x
19 15979 3 0 0.00x 0 0.00x 2 1.72x
20 24013 3 0 0.00x 0 0.00x 2 1.18x
21 16755 4 0 0.00x 0 0.00x 2 1.28x
22 17143 3 0 0.00x 0 0.00x 2 1.20x
25 23342 5 0 0.00x 0 0.00x 3 1.80x
26 20872 3 0 0.00x 0 0.00x 1 0.63x
27 23402 3 0 0.00x 0 0.00x 1 0.72x
28 19660 4 0 0.00x 0 0.00x 2 1.33x
29 17396 5 1 0.63x 0 0.00x 2 1.45x
30 20169 4 0 0.00x 0 0.00x 2 1.29x
31 22887 3 0 0.00x 0 0.00x 1 0.63x
32 20549 3 0 0.00x 0 0.00x 1 0.48x
33 17994 3 0 0.00x 0 0.00x 1 0.69x
34 15744 5 1 0.59x 0 0.00x 2 1.44x
36 18817 5 0 0.00x 0 0.00x 2 1.16x
37 17276 4 0 0.00x 0 0.00x 2 1.44x
38 18822 3 0 0.00x 0 0.00x 1 0.79x
39 15948 3 0 0.00x 0 0.00x 1 0.71x
42 22881 25 8 3.94x 0 0.00x 4 2.39x
43 19149 3 0 0.00x 0 0.00x 1 0.67x
44 20448 4 0 0.00x 0 0.00x 2 1.23x
45 20617 4 0 0.00x 0 0.00x 2 1.38x
46 19281 3 0 0.00x 0 0.00x 1 0.60x
47 18368 5 0 0.00x 0 0.00x 2 1.43x
48 19836 3 0 0.00x 0 0.00x 1 0.53x
49 25198 4 0 0.00x 0 0.00x 2 1.12x
50 24231 8 1 0.46x 0 0.00x 3 1.87x
51 21907 4 0 0.00x 0 0.00x 2 0.94x
53 17173 3 0 0.00x 0 0.00x 1 0.62x
54 23619 3 0 0.00x 0 0.00x 1 0.71x
57 19212 4 0 0.00x 0 0.00x 2 1.25x
58 18991 3 0 0.00x 0 0.00x 1 0.alignEdLib()-- WARNING: tigbgn 17343 > tigend 13556 - tiglen 13556 utgpos 17593-31149 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 31149 - tiglen 13556 utgpos 17593-31149 padding 250
alignEdLib()-- WARNING: tigbgn 15875 > tigend 13103 - tiglen 13103 utgpos 16125-29228 padding 250
alignEdLib()-- WARNING: updated tigbgn 0 > tigend 29228 - tiglen 13103 utgpos 16125-29228 padding 250
67x
60 18502 3 0 0.00x 0 0.00x 1 0.76x
61 25634 21 5 1.96x 0 0.00x 6 2.83x
62 21848 3 0 0.00x 0 0.00x 1 0.81x
64 21797 7 0 0.00x 0 0.00x 3 1.60x
65 25186 4 0 0.00x 0 0.00x 2 1.04x
66 24889 3 0 0.00x 0 0.00x 1 0.53x
67 19484 3 0 0.00x 0 0.00x 1 0.62x
68 20219 6 0 0.00x 0 0.00x 2 1.15x
69 21841 24 8 3.77x 0 0.00x 4 2.39x
71 21662 3 0 0.00x 0 0.00x 1 0.54x
72 16007 3 0 0.00x 0 0.00x 1 0.75x
73 18078 7 0 0.00x 0 0.00x 3 1.71x
74 21265 4 0 0.00x 0 0.00x 2 1.27x
75 21662 4 0 0.00x 0 0.00x 2 1.01x
78 18027 5 0 0.00x 0 0.00x 2 1.21x
79 21188 3 0 0.00x 0 0.00x 1 0.45x
80 19359 6 1 0.56x 0 0.00x 1 0.90x
81 20739 3 0 0.00x 0 0.00x 1 0.67x
83 19866 6 0 0.00x 0 0.00x 2 1.21x
85 28960 19 3 1.04x 0 0.00x 7 2.96x
86 19471 3 0 0.00x 0 0.00x 1 0.51x
87 16138 4 0 0.00x 0 0.00x 2 1.36x
88 20905 3 0 0.00x 0 0.00x 1 0.45x
89 16150 3 0 0.00x 0 0.00x 1 0.83x
90 17548 3 0 0.00x 0 0.00x 1 0.55x
93 18821 6 0 0.00x 0 0.00x 2 1.09x
94 16424 3 0 0.00x 0 0.00x 1 0.77x
96 20114 10 0 0.00x 0 0.00x 4 2.11x
97 22137 11 1 0.48x 0 0.00x 4 1.99x
98 22304 5 0 0.00x 0 0.00x 2 1.31x
100 23654 25 8 3.66x 0 0.00x 4 2.18x
101 22904 11 0 0.00x 0 0.00x 5 2.39x
102 18065 3 0 0.00x 0 0.00x 1 0.63x
103 21697 5 0 0.00x 0 0.00x 2 1.08x
104 27143 3 0 0.00x 0 0.00x 1 0.66x
106 18148 4 1 0.57x 0 0.00x 1 0.66x
107 21386 6 0 0.00x 0 0.00x 2 1.04x
109 22877 6 1 0.44x 0 0.00x 1 0.53x
110 24069 4 0 0.00x 0 0.00x 2 0.86x
111 21505 3 0 0.00x 0 0.00x 1 0.58x
112 25757 23 8 3.38x 0 0.00x 4 2.20x
116 13835 3 0 0.00x 0 0.00x 1 0.93x
117 22321 5 0 0.00x 0 0.00x 2 1.13x
119 22934 5 0 0.00x 0 0.00x 2 0.92x
120 19913 6 0 0.00x 0 0.00x 2 1.17x
121 24517 17 2 0.96x 0 0.00x 6 2.97x
122 21597 7 1 0.48x 0 0.00x 2 1.27x
123 21937 3 0 0.00x 0 0.00x 1 0.59x
125 15010 3 0 0.00x 0 0.00x 1 0.65x
126 23821 3 0 0.00x 0 0.00x 1 utgcns: utgcns/unitigConsensus.H:116: abSequence* unitigConsensus::getSequence(uint32): Assertion `id < _sequencesLen' failed.
Failed with 'Aborted'; backtrace (libbacktrace):
utility/src/system/system-stackTrace-v1.C::82 in _Z17AS_UTL_catchCrashiP9siginfo_tPv()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
utgcns/unitigConsensus.H::116 in _ZN15unitigConsensus11getSequenceEj()
utgcns/unitigConsensus.C::682 in _ZN15unitigConsensus11getSequenceEj()
utgcns/unitigConsensus.C::310 in _ZN15unitigConsensus22generateTemplateStitchEv()
utgcns/unitigConsensus.C::911 in _ZN15unitigConsensus13generatePBDAGEP5tgTigcRSt3mapIjP6sqReadSt4lessIjESaISt4pairIKjS4_EEE()
utgcns/unitigConsensus.C::1583 in _ZN15unitigConsensus8generateEP5tgTigccRSt3mapIjP6sqReadSt4lessIjESaISt4pairIKjS4_EEE()
utgcns/utgcns-processTigs.C::143 in _Z11processTigsR13cnsParameters()
utgcns/utgcns.C::406 in main()
(null)::0 in (null)()
(null)::0 in (null)()
(null)::0 in (null)()
Aborted (core dumped)
The first two errors are due to the same issue I suspect. The last one looks different but might be related if the out-of-bounds array access corrupted something else. Are you able to share the ctgStore and the seqStore for this assembly?
You can upload them to our FTP following the instructions here: https://canu.readthedocs.io/en/latest/faq.html#how-can-i-send-data-to-you
The FTP doesn't let you upload files w/duplicate file names so once your initial upload failed, there was a truncated tar.gz that blocked subsequent uploads unless you renamed them. I renamed your partial uploads so you can try again. We've definitely uploaded larger files than yours using interactive or command-line ftp. If you keep having dropped connections, can you share in another way, for example google drive or similar?
Idle, no data to reproduce. Might have been fixed by a recent commit to utgcns.