SIGSEGV at createV8Runtime() in V8 Mode on AWS

Question

SIGSEGV at createV8Runtime() in V8 Mode on AWS

Closed this issue 5 months ago · 0 comments

Problem

A test code that creates multiple Javet engines meets SIGSEGV at createV8Runtime() in V8 mode on AWS. It works well in the Node.js mode.

Environment

Host: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz, 4 cores, 7G, Amazon Linux release 2023 (Amazon Linux)
KVM: KVM virtualization detected
JVM: OpenJDK 64-Bit Server VM (17.0.9+8-LTS) for linux-amd64 JRE (17.0.9+8-LTS)
Impacted Javet Versions: v3.0.0-v3.0.2

Test Code

import com.caoccao.javet.enums.JSRuntimeType;
import com.caoccao.javet.interop.V8Runtime;
import com.caoccao.javet.interop.engine.IJavetEngine;
import com.caoccao.javet.interop.engine.JavetEnginePool;

import java.util.ArrayList;
import java.util.List;

public class TestCrash {
    public static void main(String[] args) throws Exception {
        final int threadCount;
        if (args.length > 0) {
            final String arg = args[0];
            threadCount = Integer.parseInt(arg);
            System.out.println("Using threads from args: " + threadCount);
        } else {
            threadCount = 5;
            System.out.println("Using threads from default: " + threadCount);
        }
        final List<Thread> threadList = new ArrayList<>();
        for (int i = 0; i < threadCount; i++) {
            final Thread thread = new Thread(() -> {
                try (final JavetEnginePool<V8Runtime> enginePool = new JavetEnginePool<>()) {
                    System.out.println("Created engine pool");
                    enginePool.getConfig().setJSRuntimeType(JSRuntimeType.V8);
                    try (final IJavetEngine<V8Runtime> engine = enginePool.getEngine()) {
                        // This will not be reached for threadCount >= 2
                        System.out.println("Created engine");
                    }
                } catch (Throwable t) {
                    t.printStackTrace(System.err);
                }
            });
            threadList.add(thread);
            thread.start();
        }
        for (Thread thread : threadList) {
            thread.join();
        }
        System.out.println("Quit peacefully.");
    }
}

Stack Frames

Current thread (0x00007fa054141f10):  JavaThread "Thread-3" [_thread_in_native, id=573943, stack(0x00007fa0291ff000,0x00007fa0292ff000)]

Stack: [0x00007fa0291ff000,0x00007fa0292ff000],  sp=0x00007fa0292fce00,  free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x33fe55]  v8::internal::Heap::SetUpSpaces(v8::internal::LinearAllocationArea&, v8::internal::LinearAllocationArea&)+0x845
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x2a24ef]  v8::internal::Isolate::Init(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool)+0x9af
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x2a32d9]  v8::internal::Isolate::InitWithSnapshot(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool)+0x9
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x710606]  v8::internal::Snapshot::Initialize(v8::internal::Isolate*)+0x216
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x1789b2]  v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&)+0x242
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x178bed]  v8::Isolate::New(v8::Isolate::CreateParams const&)+0x1d
C  [libjavet-v8-linux-x86_64.v.3.0.3.so+0x127fd8]  Javet::V8Runtime::CreateV8Isolate()+0x38

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.caoccao.javet.interop.V8Native.createV8Runtime(Ljava/lang/Object;)J+0
j  com.caoccao.javet.interop.V8Host.createV8Runtime(ZLcom/caoccao/javet/interop/options/RuntimeOptions;)Lcom/caoccao/javet/interop/V8Runtime;+67
j  com.caoccao.javet.interop.engine.JavetEnginePool.createEngine()Lcom/caoccao/javet/interop/engine/JavetEngine;+43
j  com.caoccao.javet.interop.engine.JavetEnginePool.getEngine()Lcom/caoccao/javet/interop/engine/IJavetEngine;+80
j  TestCrash.lambda$main$0()V+28
j  TestCrash$$Lambda$1+0x00007f9fdc000a08.run()V+0
j  java.lang.Thread.run()V+11 java.base@17.0.9
v  ~StubRoutines::call_stub

Analysis

The root cause seems that the V8 memory protection key is not properly initialized in very few VM environments with specific CPU models. The changes were committed on May 26, 2023. That causes Javet v3.0.0-v3.0.2 to have this issue.

Solution

As the SIGSEGV only occurs on Linux x86_64 with small set of CPUs, the solution is expected to be only applied to Linux x86_64 build.

1: Force the Initialization (v3.0.0-v3.0.2)

Execute the following code in the main thread during the application initialization to force V8 to initialize.

try (V8Runtime v8Runtime = V8Host.getV8Instance().createV8Runtime()) {
}

Why does this trick work?

V8 stores memory protection key flag in a global storage called ThreadIsolation. ThreadIsolation's initialization is broken in some Fedora based Linux distributions inside KVM.

A full cycle of V8Runtime ensures ThreadIsolation's initialization is performed successfully.

2: Set Environment Variable JAVET_DISABLE_PKU (v3.0.3+)

Add an environment variable JAVET_DISABLE_PKU before the application is started.

export JAVET_DISABLE_PKU=1

Why does this trick work?

V8 source code src/libplatform/default-thread-isolated-allocator.cc is patched to consume the environment variable JAVET_DISABLE_PKU so that a Linx kernel feature Memory Protection Keys is disabled.

 // found in the LICENSE file.

 #include "src/libplatform/default-thread-isolated-allocator.h"
+#include <cstdlib>

 #if V8_HAS_PKU_JIT_WRITE_PROTECT

 namespace {

 bool KernelHasPkruFix() {
+const char* env = std::getenv("JAVET_DISABLE_PKU"); if (env && std::strlen(env) > 0) { return false; }
   // PKU was broken on Linux kernels before 5.13 (see
   // https://lore.kernel.org/all/20210623121456.399107624@linutronix.de/).
   // A fix is also included in the 5.4.182 and 5.10.103 versions ("x86/fpu:

3: Patch V8 (Deprecated)

V8 code is patched to skip the unnecessary check in file src/heap/heap.cc. This is a temporary patch till V8 dev team addresses the issue officially.

   write_protect_code_memory_ = v8_flags.write_protect_code_memory;
 #if V8_HEAP_USE_PKU_JIT_WRITE_PROTECT
-  if (RwxMemoryWriteScope::IsSupported()) {
+  if (write_protect_code_memory_ && RwxMemoryWriteScope::IsSupported()) {
     // If PKU machinery is available then use it instead of conventional
     // mprotect.
     write_protect_code_memory_ = false;

Why does this trick work?

RwxMemoryWriteScope::IsSupported() is an inline function optimized by the C++ compiler. It accesses ThreadIsolation's memory protection key flag which is not initialized yet. That results in access violation that crashes the JVM.

By testing write_protect_code_memory_ prior to calling RwxMemoryWriteScope::IsSupported(), that access violation can be skipped if write_protect_code_memory_ is false. Obviously, there is no need to set write_protect_code_memory_ to false again if write_protect_code_memory_ is already false.

However, it's hard to say that RwxMemoryWriteScope::IsSupported() is supposed to be called regardless of write_protect_code_memory_ . That makes this patch argurable and deprecated. I suggest C++ developers to avoid writing such ambiguous code.