Gimbricate segfaults on a particular GFA

Question

Gimbricate segfaults on a particular GFA

RolandFaure opened this issue 4 years ago · 9 comments

We are running gimbricate on a graph produced by an assembler, with the line gimbricate -g assemblyGraph_k63.gfa -n -p h.paf -f h.fasta >h.gimbry.gfa. It segfaults after a few seconds.
When looking, it seems that it produced paf and fasta file correctly, and segfaulted after writing many sequences in h.gimbry.gfa but before writing the links.
It does not seem to come from a RAM limitation error, as the segfault comes very quickly. Other graphs outputted by the same assembler ran fine with gimbricate, thus we suppose the error might come from a particular topological feature of the gfa. Do you know of any such limitation of gimbricate ?
Thanks in advance :)

Answer 1 · 2020-08-04T07:03:13.000Z

The only thing that comes to mind is a very long overlap that causes the alignment to run out of memory. It doesn't actually eat through the memory, but attempts to allocate a lot all at once and fails. If you can share the GFA then I'd be able to confirm this. I need to use edlib to derive the alignments. This has a banded mode and can be much more memory efficient.

…

On Tue, Aug 4, 2020, 08:27 RolandFaure ***@***.***> wrote: We are running gimbricate on a graph produced by an assembler, with the line gimbricate -g assemblyGraph_k63.gfa -n -p h.paf -f h.fasta >h.gimbry.gfa. It segfaults after a few seconds. When looking, it seems that it produced paf and fasta file correctly, and segfaulted after writing many sequences in h.gimbry.gfa but before writing the links. It does not seem to come from a RAM limitation error, as the segfault comes very quickly. Other graphs outputted by the same assembler ran fine with gimbricate, thus we suppose the error might come from a particular topological feature of the gfa. Do you know of any such limitation of gimbricate ? Thanks in advance :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQEOZUKAJYFYWNEQQEMDR66S57ANCNFSM4PUBWWHA> .

Answer 2 · 2020-08-04T08:07:54.000Z

It is indeed a gfa with very long overlaps. The thing is, I am not actually using the master branch of gimbricate but the fork we made (ege-eeb fork, see issue #2 ), in which the Smith-Waterman algorithm is taken out in align.cpp, precisely to deal with large, perfect alignments (it segfaults in the same way in the ekg version of gimbricate). I checked,

gimbricate segfaulted before ever going into align.cpp.

Does that mean that gimbricate tries to allocate memory for SW before going into align.cpp ?
Or is that memory necessary for something else entirely ?

Answer 3 · 2020-08-04T08:21:44.000Z

I'll need to look at your fork. This could be caused by an error in the construction of the GFA. For instance, if edges refer to nodes that do not exist, or if the overlap that's specified is longer than one of the nodes. If you use gdb where does it show the error occuring? I do think it would be good to have edlib integrated. You can specify a percent identity, and this is used to establish a bandwidth for the alignment. You will also be able to confirm that the overlaps are exact. But it's good to know that this isn't your problem here.

…

On Tue, Aug 4, 2020, 10:08 RolandFaure ***@***.***> wrote: It is indeed a gfa with very long overlaps. The thing is, I am not actually using the master branch of gimbricate but the fork we made (ege-eeb fork, see issue #2 <#2> ), in which the Smith-Waterman algorithm is taken out in align.cpp, precisely to deal with large, perfect alignments (it segfaults in the same way in the ekg version of gimbricate). I checked, gimbricate segfaulted before ever going into align.cpp. Does that mean that gimbricate tries to allocate memory for SW before going into align.cpp ? Or is that memory necessary for something else entirely ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQENNECFPJ6ZS3CR24J3R666WRANCNFSM4PUBWWHA> .

Answer 4 · 2020-08-05T12:00:02.000Z

Using gdb, the error occurs :
#3 0x0000555555581898 in std::function<void (gfak::edge_elem const&)>::operator()(gfak::edge_elem const&) const (__args#0=..., this=0x7fffffffc5e0)
at /usr/include/c++/7/bits/std_function.h:706
#4 gfak::GFAKluge::for_each_edge_line_in_file(char*, std::function<void (gfak::edge_elem const&)>) (this=0x7fffffffc370, filename=,
func=...) at gimbricate/deps/gfakluge/src/gfakluge.hpp:936
#5 0x0000555555568d17 in main (argc=, argv=)
at gimbricate/src/main.cpp:132

I've checked the GFA and found no overlaps longer than an edge or edges referring to non-existing nodes. Moreover all other GFA produced by the same assembler are processed without problem.

Answer 5 · 2020-08-05T13:38:41.000Z

This indicates an error when parsing an edge record. Would you please share the GFA? I'll need to take a look at it.

…

On Wed, Aug 5, 2020, 14:00 RolandFaure ***@***.***> wrote: Using gdb, the error occurs : #3 <#3> 0x0000555555581898 in std::function<void (gfak::edge_elem const&)>::operator()(gfak::edge_elem const&) const (__args#0=..., this=0x7fffffffc5e0) at /usr/include/c++/7/bits/std_function.h:706 #4 <#4> gfak::GFAKluge::for_each_edge_line_in_file(char*, std::function<void (gfak::edge_elem const&)>) (this=0x7fffffffc370, filename=, func=...) at gimbricate/deps/gfakluge/src/gfakluge.hpp:936 #5 0x0000555555568d17 in main (argc=, argv=) at gimbricate/src/main.cpp:132 I've checked the GFA and found no overlaps longer than an edge or edges referring to non-existing nodes. Moreover all other GFA produced by the same assembler are processed without problem. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQEMOBTCBQDJYOMUAW4LR7FCVDANCNFSM4PUBWWHA> .

Answer 6 · 2020-08-05T14:19:32.000Z

https://mega.nz/file/eYdx3YzK#ZwG8uIRGMfZpYHB-ZS7Y9jFK8nJOTdAGO9VHWTHaxSk

Thank you very much for your time :)

Answer 7 · 2020-08-06T09:12:54.000Z

Would you have another mechanism to transfer the file rather than mega? It's requiring that I create an account.

…

On Wed, Aug 5, 2020 at 4:19 PM RolandFaure ***@***.***> wrote: https://mega.nz/file/eYdx3YzK#ZwG8uIRGMfZpYHB-ZS7Y9jFK8nJOTdAGO9VHWTHaxSk Thank you very much for your time :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQELQVOD4J7NBWGYYNCTR7FTALANCNFSM4PUBWWHA> .

Answer 8 · 2020-08-06T09:20:21.000Z

On wetransfer
https://we.tl/t-ZmTt1De3aM

Answer 9 · 2020-08-30T09:27:28.000Z

@RolandFaure the problem here was that your GFA file didn't have a header. Thus, the first record was skipped, but the link records associated with it were read. This caused the segfault.

(echo 'H\tVN:Z:1.0' ; cat assemblyGraph_k63.gfa) >assemblyGraph_k63.fixed.gfa
gimbricate -g assemblyGraph_k63.fixed.gfa -p assemblyGraph_k63.paf -f assemblyGraph_k63.fa -t 16 >assemblyGraph_k63.gimbry.gfa

This runs without issue (at least on the edlib-overlaps branch).

I've pushed a fix that will check that the header is present and the file is apparently GFAv1.