Overview
jvmkill is a simple JVMTI agent that forcibly terminates the JVM
when it is unable to allocate memory or create a thread. This is important
for reliability purposes: an OutOfMemoryError will often leave the JVM
in an inconsistent state. Terminating the JVM will allow it to be restarted
by an external process manager.
It is often useful to automatically dump the Java heap using the
-XX:+HeapDumpOnOutOfMemoryError JVM argument. This agent will be
notified and terminate the JVM after the heap dump completes.
A common alternative to this agent is to use the
-XX:OnOutOfMemoryError JVM argument to execute a kill -9 command.
Unfortunately, the JVM uses the fork() system call to execute the kill
command and that system call can fail for large JVMs due to memory
overcommit limits in the operating system. This is the problem that
motivated the development of this agent.
Pre-requisites
Install Rust v1.19.0 or later.
Ensure that you have libclang v3.9 or later installed. To install this on Ubuntu Trusty see ask ubuntu
and set $LIBCLANG_PATH to the directory containing (v3.9 or later of) libclang.so and libclang.so.1.
Building
To build the agent, install the above pre-requisites, then issue:
cargo build --release -p jvmkill
If the build fails, update the dependencies as follows then try again:
cargo update
Testing
To run the tests, install the above pre-requisites, then issue:
cargo test --all
Usage
Run Java with the agent added as a JVM argument:
-agentpath:/path/to/libjvmkill.so=<parameters>
Alternatively, if modifying the Java command line is not possible, the
above may be added to the JAVA_TOOL_OPTIONS environment variable.
Agent parameters
The agent configurations can be passed using the standard agent mechanism. The parameters should be passed as a comma separated string. Eg.: count=2,time=10 The agent accepts the following parameters:
count
Configures the limit of resourceExhausted events that can be fired in the configured time interval. Defaults to 0 if not provided (JVM is killed with a single fired event).
time
Configures the time limit (in seconds) in which resourceExhausted events are kept in the counter. Defaults to 1 if not provided.
heapDumpPath
Configures a file to which a heap dump is written before the agent kills the JVM.
To enable heap dump generation, set the parameter to the path of a file in a writable directory with sufficient free space. The path may be absolute or relative to the working directory where the JVM was started. If the parameter is not specified, no heap dump is generated.
The path is treated as a strftime format specification, although the precise set of format codes supported depends on the platform. For example, the string "%a-%d-%b-%Y-%T-%z" approximates the date format of RFC 2822 while avoiding embedded spaces (which are awkward in agent parameters).
A heap dump (of live objects only) is generated if a path is specified and the HotSpot Diagnostic MXBean is available. If the parent directories of the path do not exist, they are created.
If the file exists before the heap dump is produced, it is overwritten. Including the date and time in the file path, using strftime format codes, may reduce the risk that the file already exists.
printHeapHistogram
Determines whether or not a histogram of heap usage is printed before the agent kills the JVM. To enable histogram printing, set the parameter to 1. Defaults to 0 (disabled) if not provided.
Each entry in the histogram describes the number of instances of a particular Java type, the total number of bytes in the heap consumed by those instances, and the name of the type.
The histogram is sorted in order of decreasing total number of bytes.
The histogram may be truncated. To set the number of entries that appear, use the heapHistogramMaxEntries parameter.
heapHistogramMaxEntries
When histogram printing is enabled, limits the number of entries in the histogram to the value of the parameter. Defaults to 100 if not provided. Set the parameter to 0 to print the entire histogram.
printMemoryUsage
Determines whether or not memory usage is printed before the agent kills the JVM. To disable memory usage printing, set the parameter to 0. Defaults to 1 (enabled) if not provided.
If the agent has been driven because the JVM is unable to create a thread, memory usage is not printed as attempting to obtain memory usage statistics can cause the agent to fail in which case the JVM is not killed.
When testing thread exhaustion with a small heap on Linux, it was found that the agent can be driven for heap exhaustion and yet obtaining memory usage stats can still cause the agent to fail in which case the JVM is not killed. If this is encountered with a real application, printing memory usage can be disabled.
License
The jvmkill agent is Open Source software released under the Apache 2.0 license.
Development
Please refer to the Developers' Guide.