Getting Segfault when running ReorderSam via GATK
Closed this issue · 1 comments
rickymagner commented
Bug Report
Affected tool(s)
ReorderSam
Affected version(s)
2.27.5 inside GATK v4.3.0.0
Description
I ran gatk ReorderSam on a bam which seemed to be running fine for quite a while. Once it went through writing all of the reads for each contigs, the JVM complained about a segfault. The full debug log is long, but here is the beginning (which is hopefully most relevant):
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f7616c507bd, pid=124452, tid=0x00007f75fa9fd700
#
# JRE version: OpenJDK Runtime Environment (8.0_345-b01) (build 1.8.0_345-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.345-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x9b57bd] oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x7d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%207&component=java-1.8.0-openjdk
#
--------------- T H R E A D ---------------
Current thread (0x00007f761006e000): GCTaskThread [stack: 0x00007f75fa8fe000,0x00007f75fa9fe000] [id=124468]
siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000
Registers:
RAX=0x001e882181481e1c, RBX=0x00000007d6f4b3e0, RCX=0x00000000c0100800, RDX=0x0000000000000000
RSP=0x00007f75fa9fcc30, RBP=0x00007f75fa9fccb0, RSI=0xfffffffffffffff8, RDI=0x0000000000000017
R8 =0x0000000000000001, R9 =0x00007f76174c4910, R10=0x0000000000001002, R11=0x00007f76174fcf90
R12=0x000000008428b858, R13=0x001e882181481e1e, R14=0x00007f76103af880, R15=0x0000000000001002
RIP=0x00007f7616c507bd, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
TRAPNO=0x000000000000000d
Top of Stack: (sp=0x00007f75fa9fcc30)
0x00007f75fa9fcc30: 00007f761006f328 0000000000000078
0x00007f75fa9fcc40: 00007f75fa9fcca0 0000000000001c30
0x00007f75fa9fcc50: 0000000000000001 00007f7600000001
0x00007f75fa9fcc60: 000000008428a720 000000008428b060
0x00007f75fa9fcc70: 00007f75fa9fcd40 3f791eb3fcd46400
0x00007f75fa9fcc80: 000000008428b060 00007f76103af880
0x00007f75fa9fcc90: 000000008428b858 0000000000000001
0x00007f75fa9fcca0: 00007f76103af950 00007f7617312930
0x00007f75fa9fccb0: 00007f75fa9fcd30 00007f7616c50168
0x00007f75fa9fccc0: 00000001fa9fcd30 00007f76103af920
0x00007f75fa9fccd0: 008a3a3b00006c65 000000008428b060
0x00007f75fa9fcce0: 008a3a3c00006c65 0000000000000400
0x00007f75fa9fccf0: 00000001fa9fcd30 3f791eb3fcd46400
0x00007f75fa9fcd00: 0000000480f00000 0000000000000008
0x00007f75fa9fcd10: 00007f75fa9fcd60 00007f75fa9fcd70
0x00007f75fa9fcd20: 0000000000000000 00007f76103b1600
0x00007f75fa9fcd30: 00007f75fa9fcdb0 00007f7616c5466a
0x00007f75fa9fcd40: 00007f7617312930 0000000081256024
0x00007f75fa9fcd50: 00007f7534453650 00007f76103af880
0x00007f75fa9fcd60: 00007f757a98d64d 00007f7616be44d1
0x00007f75fa9fcd70: 0000000081256025 3f791eb3fcd46400
0x00007f75fa9fcd80: 00007f75fa9fcdb0 0000000000000001
0x00007f75fa9fcd90: 00007f7534453650 00007f7616e95be4
0x00007f75fa9fcda0: 00007f761006e000 00007f76173318c8
0x00007f75fa9fcdb0: 00007f75fa9fced0 00007f761687c082
0x00007f75fa9fcdc0: 0000000000000000 00007f7617312510
0x00007f75fa9fcdd0: 00000000000003d8 00007f761006ebb8
0x00007f75fa9fcde0: 00007f761006e7e0 00007f75fa9fce60
0x00007f75fa9fcdf0: 00007f761006e790 00007f761006e7d0
0x00007f75fa9fce00: 00007f75fa9fce10 0000000000000000
0x00007f75fa9fce10: 0000000000000000 00007f761006f300
0x00007f75fa9fce20: 00007f761006e000 00007f761006ebc0
Instructions: (pc=0x00007f7616c507bd)
0x00007f7616c5079d: 01 0f 85 4c 03 00 00 89 c8 c1 f8 03 41 f6 c5 01
0x00007f7616c507ad: 4c 63 f8 0f 85 4a 02 00 00 4c 89 e8 48 83 e0 fd
0x00007f7616c507bd: 4c 8b 18 48 8d 05 a9 f4 6e 00 49 c1 eb 03 41 83
0x00007f7616c507cd: e3 0f 44 3b 18 0f 82 43 02 00 00 4d 8b 66 30 4a
Register to memory mapping:
RAX=0x001e882181481e1c is an unknown value
RBX=0x00000007d6f4b3e0 is an oop
[B
- klass: {type array byte}
- length: 32768
RCX=
[error occurred during error reporting (printing register info), id 0xb]
Stack: [0x00007f75fa8fe000,0x00007f75fa9fe000], sp=0x00007f75fa9fcc30, free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x9b57bd] oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x7d
V [libjvm.so+0x9b5168] PSPromotionManager::drain_stacks_depth(bool)+0x6f8
V [libjvm.so+0x9b966a] StealTask::do_it(GCTaskManager*, unsigned int)+0x19a
V [libjvm.so+0x5e1082] GCTaskThread::run()+0x1a2
V [libjvm.so+0x9403c2] java_start(Thread*)+0xf2
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f74fc001000 JavaThread "ForkJoinPool.commonPool-worker-3" daemon [_thread_blocked, id=45851, stack(0x00007f757a70a000,0x00007f757a80b000)]
0x00007f749c001000 JavaThread "ForkJoinPool.commonPool-worker-6" daemon [_thread_blocked, id=45582, stack(0x00007f757b114000,0x00007f757b215000)]
0x00007f74c8001000 JavaThread "ForkJoinPool.commonPool-worker-20" daemon [_thread_blocked, id=44931, stack(0x00007f757b518000,0x00007f757b619000)]
0x00007f74fc003000 JavaThread "ForkJoinPool.commonPool-worker-15" daemon [_thread_blocked, id=44378, stack(0x00007f757a104000,0x00007f757a205000)]
0x00007f74dc003000 JavaThread "ForkJoinPool.commonPool-worker-4" daemon [_thread_blocked, id=43581, stack(0x00007f75801e2000,0x00007f75802e3000)]
0x00007f7500001000 JavaThread "ForkJoinPool.commonPool-worker-27" daemon [_thread_blocked, id=42684, stack(0x00007f757aa0d000,0x00007f757ab0e000)]
0x00007f74bc002000 JavaThread "ForkJoinPool.commonPool-worker-24" daemon [_thread_blocked, id=42127, stack(0x00007f757ae11000,0x00007f757af12000)]
0x00007f74b0001000 JavaThread "ForkJoinPool.commonPool-worker-21" daemon [_thread_blocked, id=41322, stack(0x00007f757a609000,0x00007f757a70a000)]
0x00007f74bc001000 JavaThread "ForkJoinPool.commonPool-worker-28" daemon [_thread_blocked, id=41053, stack(0x00007f757af12000,0x00007f757b013000)]
0x00007f74f8003000 JavaThread "ForkJoinPool.commonPool-worker-26" daemon [_thread_blocked, id=40196, stack(0x00007f757ac0f000,0x00007f757ad10000)]
0x00007f74a4001000 JavaThread "ForkJoinPool.commonPool-worker-13" daemon [_thread_blocked, id=38975, stack(0x00007f757a80b000,0x00007f757a90c000)]
0x00007f7508005000 JavaThread "ForkJoinPool.commonPool-worker-22" daemon [_thread_blocked, id=38602, stack(0x00007f757b013000,0x00007f757b114000)]
0x00007f74c0004800 JavaThread "ForkJoinPool.commonPool-worker-29" daemon [_thread_blocked, id=38274, stack(0x00007f757b619000,0x00007f757b71a000)]
0x00007f74ac001000 JavaThread "ForkJoinPool.commonPool-worker-23" daemon [_thread_blocked, id=37074, stack(0x00007f757a205000,0x00007f757a306000)]
0x00007f7508001000 JavaThread "ForkJoinPool.commonPool-worker-30" daemon [_thread_blocked, id=28008, stack(0x00007f75802e3000,0x00007f75803e4000)]
0x00007f74e4001000 JavaThread "ForkJoinPool.commonPool-worker-16" daemon [_thread_blocked, id=24229, stack(0x00007f757a306000,0x00007f757a407000)]
0x00007f74a8001000 JavaThread "ForkJoinPool.commonPool-worker-9" daemon [_thread_blocked, id=8881, stack(0x00007f757ad10000,0x00007f757ae11000)]
0x00007f761041e800 JavaThread "Service Thread" daemon [_thread_blocked, id=124503, stack(0x00007f75903c4000,0x00007f75904c5000)]
0x00007f7610411800 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=124502, stack(0x00007f75904c6000,0x00007f75905c6000)]
0x00007f761040f000 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=124501, stack(0x00007f75905c7000,0x00007f75906c7000)]
0x00007f761040d000 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=124500, stack(0x00007f75906c8000,0x00007f75907c8000)]
0x00007f761040a800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=124499, stack(0x00007f75907c9000,0x00007f75908c9000)]
0x00007f7610408000 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=124498, stack(0x00007f75908ca000,0x00007f75909ca000)]
0x00007f7610406000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=124497, stack(0x00007f75909cb000,0x00007f7590acb000)]
0x00007f7610403800 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=124496, stack(0x00007f7590acc000,0x00007f7590bcc000)]
0x00007f7610401800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=124495, stack(0x00007f7590bcd000,0x00007f7590ccd000)]
0x00007f76103ff000 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=124494, stack(0x00007f7590cce000,0x00007f7590dce000)]
0x00007f76103fd000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=124493, stack(0x00007f7590dcf000,0x00007f7590ecf000)]
0x00007f76103fb000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=124492, stack(0x00007f7590ed0000,0x00007f7590fd0000)]
0x00007f76103f8000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=124491, stack(0x00007f7590fd1000,0x00007f75910d1000)]
0x00007f76103f4800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=124490, stack(0x00007f75910d1000,0x00007f75911d2000)]
0x00007f76103c2800 JavaThread "Finalizer" daemon [_thread_blocked, id=124489, stack(0x00007f75911d2000,0x00007f75912d3000)]
0x00007f76103bd800 JavaThread "Reference Handler" daemon [_thread_blocked, id=124488, stack(0x00007f75912d3000,0x00007f75913d4000)]
0x00007f761004c000 JavaThread "main" [_thread_blocked, id=124457, stack(0x00007f7617e82000,0x00007f7617f82000)]
Other Threads:
0x00007f76103b4000 VMThread [stack: 0x00007f75913d5000,0x00007f75914d5000] [id=124487]
0x00007f7610421000 WatcherThread [stack: 0x00007f75902c4000,0x00007f75903c4000] [id=124504]
=>0x00007f761006e000 (exited) GCTaskThread [stack: 0x00007f75fa8fe000,0x00007f75fa9fe000] [id=124468]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007f7610049340] Threads_lock - owner thread: 0x00007f76103b4000
[0x00007f7610049840] Heap_lock - owner thread: 0x00007f761004c000
heap address: 0x0000000080200000, size: 30718 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
Compressed class space size: 1073741824 Address: 0x0000000800000000
Heap:
PSYoungGen total 9725952K, used 9283247K [0x0000000580100000, 0x00000007fda00000, 0x0000000800000000)
eden space 9020416K, 100% used [0x0000000580100000,0x00000007a6a00000,0x00000007a6a00000)
from space 705536K, 37% used [0x00000007d2900000,0x00000007e29abff0,0x00000007fda00000)
to space 712704K, 0% used [0x00000007a6a00000,0x00000007a6a18010,0x00000007d2200000)
ParOldGen total 2380800K, used 214269K [0x0000000080200000, 0x0000000111700000, 0x0000000580100000)
object space 2380800K, 9% used [0x0000000080200000,0x000000008d3476f0,0x0000000111700000)
Metaspace used 43356K, capacity 43736K, committed 44160K, reserved 1087488K
class space used 5844K, capacity 5965K, committed 6016K, reserved 1048576K
Card table byte_map: [0x00007f75fd400000,0x00007f7601000000] byte_map_base: 0x00007f75fcfff000
Marking Bits: (ParMarkBitMap*) 0x00007f761733f6c0
Begin Bits: [0x00007f75a0010000, 0x00007f75be008000)
End Bits: [0x00007f75be008000, 0x00007f75dc000000)
Polling page: 0x00007f7617f94000
CodeCache: size=245760Kb used=10945Kb max_used=12843Kb free=234814Kb
bounds [0x00007f7601000000, 0x00007f7601cb0000, 0x00007f7610000000]
total_blobs=3053 nmethods=2569 adapters=393
compilation: enabled
0x00007f74c8001000 JavaThread "ForkJoinPool.commonPool-worker-20" daemon [_thread_blocked, id=44931, stack(0x00007f757b518000,0x00007f757b619000)]
0x00007f74fc003000 JavaThread "ForkJoinPool.commonPool-worker-15" daemon [_thread_blocked, id=44378, stack(0x00007f757a104000,0x00007f757a205000)]
0x00007f74dc003000 JavaThread "ForkJoinPool.commonPool-worker-4" daemon [_thread_blocked, id=43581, stack(0x00007f75801e2000,0x00007f75802e3000)]
0x00007f7500001000 JavaThread "ForkJoinPool.commonPool-worker-27" daemon [_thread_blocked, id=42684, stack(0x00007f757aa0d000,0x00007f757ab0e000)]
0x00007f74bc002000 JavaThread "ForkJoinPool.commonPool-worker-24" daemon [_thread_blocked, id=42127, stack(0x00007f757ae11000,0x00007f757af12000)]
0x00007f74b0001000 JavaThread "ForkJoinPool.commonPool-worker-21" daemon [_thread_blocked, id=41322, stack(0x00007f757a609000,0x00007f757a70a000)]
0x00007f74bc001000 JavaThread "ForkJoinPool.commonPool-worker-28" daemon [_thread_blocked, id=41053, stack(0x00007f757af12000,0x00007f757b013000)]
0x00007f74f8003000 JavaThread "ForkJoinPool.commonPool-worker-26" daemon [_thread_blocked, id=40196, stack(0x00007f757ac0f000,0x00007f757ad10000)]
0x00007f74a4001000 JavaThread "ForkJoinPool.commonPool-worker-13" daemon [_thread_blocked, id=38975, stack(0x00007f757a80b000,0x00007f757a90c000)]
0x00007f7508005000 JavaThread "ForkJoinPool.commonPool-worker-22" daemon [_thread_blocked, id=38602, stack(0x00007f757b013000,0x00007f757b114000)]
0x00007f74c0004800 JavaThread "ForkJoinPool.commonPool-worker-29" daemon [_thread_blocked, id=38274, stack(0x00007f757b619000,0x00007f757b71a000)]
0x00007f74ac001000 JavaThread "ForkJoinPool.commonPool-worker-23" daemon [_thread_blocked, id=37074, stack(0x00007f757a205000,0x00007f757a306000)]
0x00007f7508001000 JavaThread "ForkJoinPool.commonPool-worker-30" daemon [_thread_blocked, id=28008, stack(0x00007f75802e3000,0x00007f75803e4000)]
0x00007f74e4001000 JavaThread "ForkJoinPool.commonPool-worker-16" daemon [_thread_blocked, id=24229, stack(0x00007f757a306000,0x00007f757a407000)]
0x00007f74a8001000 JavaThread "ForkJoinPool.commonPool-worker-9" daemon [_thread_blocked, id=8881, stack(0x00007f757ad10000,0x00007f757ae11000)]
0x00007f761041e800 JavaThread "Service Thread" daemon [_thread_blocked, id=124503, stack(0x00007f75903c4000,0x00007f75904c5000)]
0x00007f7610411800 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=124502, stack(0x00007f75904c6000,0x00007f75905c6000)]
0x00007f761040f000 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=124501, stack(0x00007f75905c7000,0x00007f75906c7000)]
0x00007f761040d000 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=124500, stack(0x00007f75906c8000,0x00007f75907c8000)]
0x00007f761040a800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=124499, stack(0x00007f75907c9000,0x00007f75908c9000)]
0x00007f7610408000 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=124498, stack(0x00007f75908ca000,0x00007f75909ca000)]
0x00007f7610406000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=124497, stack(0x00007f75909cb000,0x00007f7590acb000)]
0x00007f7610403800 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=124496, stack(0x00007f7590acc000,0x00007f7590bcc000)]
0x00007f7610401800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=124495, stack(0x00007f7590bcd000,0x00007f7590ccd000)]
0x00007f76103ff000 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=124494, stack(0x00007f7590cce000,0x00007f7590dce000)]
0x00007f76103fd000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=124493, stack(0x00007f7590dcf000,0x00007f7590ecf000)]
0x00007f76103fb000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=124492, stack(0x00007f7590ed0000,0x00007f7590fd0000)]
0x00007f76103f8000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=124491, stack(0x00007f7590fd1000,0x00007f75910d1000)]
0x00007f76103f4800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=124490, stack(0x00007f75910d1000,0x00007f75911d2000)]
0x00007f76103c2800 JavaThread "Finalizer" daemon [_thread_blocked, id=124489, stack(0x00007f75911d2000,0x00007f75912d3000)]
0x00007f76103bd800 JavaThread "Reference Handler" daemon [_thread_blocked, id=124488, stack(0x00007f75912d3000,0x00007f75913d4000)]
0x00007f761004c000 JavaThread "main" [_thread_blocked, id=124457, stack(0x00007f7617e82000,0x00007f7617f82000)]
Steps to reproduce
This might be tough since it's unclear if it's specific to my input bam or not.
Expected behavior
It would finish writing the bam file.
Actual behavior
It writes most of the bam file from what I can tell, but then doesn't include the EOF token since it crashes right at the end.
I'm going to try rerunning with a later version of GATK to see if that helps. I'm not sure this is reasonable for anyone else to try debugging, but thought I'd post it in case someone has seen something similar and has some ideas what might be happening.
rickymagner commented
I tried rerunning with GATK 4.5.0.0 which has Picard 3.1.1 and HTSJDK 4.1.0 and it finished successfully. I'm going to assume this was either an old version bug or a very weird transient edge case. I'm not invested enough to try to rerun it and see which of the two it was, but feel free to reopen if anyone wants to investigate this further.