krashpad

A command line tool to parse Java fatal error logs.

Supports

OpenJDK derivatives:

Adoptium/AdoptOpenJDK
Azul
Microsoft build of OpenJDK
Oracle JDK
Red Hat build of OpenJDK
etc.

Running

docker/podman

Fastest/easiest way to run the latest version. The image in github.io is built/updated with every commit to the main branch.

Run the following in the directory where hs_err_pid12345 exists:

$ docker run --pull=always -v "$PWD":/home/krashpad/files:z ghcr.io/mgm3746/krashpad:main -c /home/krashpad/files/hs_err_pid12345.log > hs_err_pid12345.log.pad

Name the analysis report after the log file with a ".pad" extension (or whatever convention you want to use).

jar

Check out the main branch and build it.

Get source:

git clone https://github.com/mgm3746/krashpad.git

Build it:

cd krashpad
mvn clean (rebuilding)
mvn --settings settings.xml package
mvn --settings settings.xml javadoc:javadoc

Run it:

java -jar krashpad-LATEST.jar /path/to/hs_err_pid12345

Usage

usage: krashpad [OPTION]... [FILE]
 -c,--console        print report to stdout instead of file
 -h,--help           help
 -o,--output <arg>   output file name (default report.txt)

Notes:

The custom output file name option only applies when running as a jar. It is useful when analyzing multiple fatal error logs.

Report

The report below is based on the following fatal error log:

https://github.com/mgm3746/krashpad/blob/main/src/test/data/dataset23.txt

$ cd /path/to/krashpad/src/test/data/
$ docker run --pull=always -v "$PWD":/home/krashpad/files:z ghcr.io/mgm3746/krashpad:main -o mgm.txt -c /home/krashpad/files/dataset23.txt > dataset23.txt.pad

A file called report.txt is created in the directory where the krashpad tool is run with analysis identifying the issue:

dataset23.txt
========================================
OS:
----------------------------------------
Version: Red Hat Enterprise Linux Server release 7.7 (Maipo)
ARCH: X86_64
CPUs (cpu x cpu cores x hyperthreading): 6
Memory: 25929M
Memory Free: 1500M (6%)
Memory Available: 18164M (70%)
Swap: 7632M
Swap Free: 7626M (100%)
========================================
JVM:
----------------------------------------
RPM: java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64
Vendor: RED_HAT
VM State: not at safepoint (normal execution)
Crash Date: Tue Dec  1 14:30:22 2020 KST
Run Time: 0d 0h 13m 8s
Garbage Collectors: PARALLEL_SCAVENGE, PARALLEL_OLD
Heap Max: 1024M
Heap Allocation: 972M (95% Heap Max)
Heap Used: 513M (53% Heap Allocation)
Heap Starting Address: 3072M
Compressed oops mode: BIT32
Metaspace Max: 256M
Metaspace Allocation: 84M (33% Metaspace Max)
Metaspace Used: 77M (92% Metaspace Allocation)
Thread Stack Size: 1024K
# Java threads: 2
Thread Stack Memory: 2M
Code Cache Max: 420M
JVM Memory Max: >1702M (7% OS Memory)
========================================
Application:
----------------------------------------
ID: UNKNOWN
JVM Args: -Dcall_P203 -Xms1024m -Xmx1024m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:/path/to/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/  -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput -XX:LogFile=/path/to/jvm.log 
========================================
Threads:
----------------------------------------
Current thread: JavaThread "domain-10" daemon [_thread_in_native, id=25945, stack(0x00007fa1bf03b000,0x00007fa1bf13c000)]
========================================
Errors:
----------------------------------------
#  SIGSEGV (0xb) at pc=0x00007fa2a4353667, pid=25779, tid=0x00007fa1bf13b700
# C  [libc.so.6+0x14d667]  __memcpy_ssse3+0xb57
========================================
Stack:
----------------------------------------
Stack: [0x00007fa1bf03b000,0x00007fa1bf13c000],  sp=0x00007fa1bf138608,  free space=1013k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x14d667]  __memcpy_ssse3+0xb57
C  [libzip.so+0x6034]  ZIP_GetEntry2+0xf4
C  [libzip.so+0x3c1d]  Java_java_util_zip_ZipFile_getEntry+0xfd
J 302  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007fa287303dce [0x00007fa287303d00+0xce]
J 473 C2 java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (22 bytes) @ 0x00007fa2873a5e94 [0x00007fa2873a5b40+0x354]
J 4848 C1 jeus.deploy.archivist.InputJarArchive.getEntry(Ljava/lang/String;)Ljava/io/InputStream; (104 bytes) @ 0x00007fa28763726c [0x00007fa2876371c0+0xac]
J 14034 C1 jeus.service.archive.ArchiveClassLoader.findClass0(Ljava/lang/String;Ljeus/deploy/archivist/AbstractArchive;)Ljava/lang/Class; (206 bytes) @ 0x00007fa28963fbfc [0x00007fa28963f5e0+0x61c]
j  jeus.servlet.loader.ContextLoader.findClass0(Ljava/lang/String;Ljeus/deploy/archivist/AbstractArchive;)Ljava/lang/Class;+3
...
========================================
ANALYSIS:
----------------------------------------
error
----------------------------------------
*There is an application or operations invalid use case resulting in an attempt to modify a file while Java has it open. A workaround to avoid the crash is to disable memory mapping in ZipFile with -Dsun.zip.disableMemoryMapping=true (at the cost of some loss in performance), or upgrade to JDK 11. Backporting a fix to JDK8 would not address the root cause, and the OpenJDK developers have determined it is too risky. References: (1) https://access.redhat.com/solutions/6983666. (2) https://bugs.openjdk.org/browse/JDK-8145260.
----------------------------------------
warn
----------------------------------------
*JDK is not the latest version. Latest version is 1.8.0_362-b08 (newer by 16 versions and 1332 days).
*Unidentified log line(s). Please submit an issue so we can investigate: https://github.com/mgm3746/krashpad/issues. If attaching a fatal error log, be sure to review it and remove any sensitive information.
*The fatal error log is very old (>30 days).
*MaxMetaspaceSize < CompressedClassSpaceSize, resulting in the JVM adjusting down the Class Metadata and Compressed Class Space sizes as follows: CompressedClassSpaceSize' = MaxMetaspaceSize(256M) - [2 * InitialBootClassLoaderMetaspaceSize(4M)] = 248M. Class Metadata Size' = MaxMetaspaceSize(256M) - CompressedClassSpaceSize'(248M) = 8M.
*The gc log file has a static name and will be overwritten on JVM startup. Enable log file rotation and/or include process id or datestamp in the file name (e.g. -Xloggc:gc_%p_%t.log).
*Consider enabling gc log file rotation (-XX:+UseGCLogFileRotation -XX:GCLogFileSize=N[K|M|G] -XX:NumberOfGCLogFiles=N) to protect disk space.
----------------------------------------
info
----------------------------------------
*Red Hat build of OpenJDK rpm install.
*The JDK is very old (3.9 years). Has the application been running without issue in production for a long time? Has something changed recently (e.g. application upgrade, load, etc.) that might have triggered the issue?
*Signal number SIGSEGV: Segmentation fault. Accessing valid memory in an invalid way.
*Initial and/or max metaspace size is set. This is generally not recommended. Reference: https://access.redhat.com/solutions/1489263.
*Metaspace(256M) = Class Metadata(8M) + Compressed Class Space(248M).
*The -XX:+PrintHeapAtGC option is causing additional heap information to be output in the gc log. The additional data is not typically used for gc analysis. If there is not a good use case for enabling this option, remove it to reduce gc logging overhead.
*Diagnostic options. The following should be removed when relevant troubleshooting is completed, as they add additional overhead and are not recommended/supported for general production use: -XX:+UnlockDiagnosticVMOptions -XX:+LogVMOutput.
========================================
1 UNIDENTIFIED LOG LINES:
----------------------------------------
MGM was here!
========================================

Notes:

The report contains nine sections: (1) OS, (2) Container, (3) JVM, (4) Application, (5) Threads, (6) Errors, (7) Stack, (8) ANALYSIS, and (9) UNIDENTIFIED LOG LINES.
Some sections will only be displayed if relevant (e.g. Container if a container environment is identified).
There is a limit of 1000 unidentified log lines that will be reported.
Please report unidentified log lines by opening an issue: https://github.com/mgm3746/krashpad/issues. Attach the fatal error log after reviewing it and removing any sensitive information.

Copyright

All rights reserved. This program and the accompanying materials are made available under the terms of the Eclipse Public License - v 2.0 which accompanies this distribution, and is available at https://www.eclipse.org/legal/epl-2.0/.