java-faas-benchmarks

This repo includes a harness and small set of benchmarks for comparing performance between OpenJDK and TeaVM/Wasmtime in a multi-tenant fuction-as-a-service (FaaS) environment. In such an environment, each invocation of a function must be isolated from others to ensure security and statelessness.

Each case is benchmarked using four different strategies:

Using OpenJDK's JNI Invocation API. Each invocation reuses the same JVM.
As above, but forking a new process for each invocation.
Using GraalVM's native image compiler, forking a new process for each invocation.
Using TeaVM-WASI to target WebAssembly, which is run in Wasmtime. Each invocation uses a new module instance.

Running the benchmarks

Prerequisites

POSIX-compatible OS (due to the use of the fork system call)
Rust
A recent OpenJDK (e.g. apt install openjdk-18-jdk-headless)
GraalVM (I used GraalVM CE 21.3.3.1)
Maven (e.g. apt install maven)
(Optional) wasm-opt (e.g. apt install binaryen)

Build the test cases

Adjust GRAALVM_HOME according to where you've installed it.

export GRAALVM_HOME=/opt/graalvm-ce-java17-21.3.3.1
export JAVA_HOME=$GRAALVM_HOME
export PATH=$GRAALVM_HOME/bin:$PATH
(cd apps && mvn -Pnative -DskipTests package)
# optionally run `wasm-opt`:
for x in mandelbrot nbody pidigits spectralnorm simple hello; do
  wasm-opt -O3 -s 0 --strip apps/$x/target/generated/wasm/teavm-wasm/classes.wasm -o $x.wasm
  mv $x.wasm apps/$x/target/generated/wasm/teavm-wasm/classes.wasm
done

Run the benchmarks

Adjust LD_LIBRARY_PATH according to where libjvm.so is in your OpenJDK installation.

LD_LIBRARY_PATH=/usr/lib/jvm/java-18-openjdk-amd64/lib/server/ cargo +nightly bench

Note that you may want to run the jvm_fork_* tests separately since performance varies considerably when they are run as a group:

export LD_LIBRARY_PATH=/usr/lib/jvm/java-18-openjdk-amd64/lib/server/
for test in hello mandelbrot nbody pidigits simple spectralnorm; do
  cargo +nightly bench jvm_fork_$test;
done

Sample results (Ubuntu 22.04, OpenJDK 18, GraalVM 21, Intel Core i7-7600U CPU @ 2.80GHz, 16GB RAM)