Native image crashes with NullPointerException on Kubernetes (probably GraalVM bug with workaround)

Question

Native image crashes with NullPointerException on Kubernetes (probably GraalVM bug with workaround)

ndwinton opened this issue 2 years ago · 8 comments

Under Spring Boot 3.0.2, native-compiled container image running on Kubernetes (in this case a "vanilla" 1.24.7 3-node cluster) fails with a NullPointerException. The same image runs perfectly directly under Docker.

After some research, I believe the underlying cause to be a current GraalVM bug. As per that issue, there is an available workaround (see below) which could be incorporated into the build process pending resolution of the underlying issue.

Steps to reproduce

# Generate simplest web app
curl https://start.spring.io/starter.zip -d dependencies=web,native -d bootVersion=3.0.2.RELEASE -d baseDir=demo -o demo.zip

# Unpack and build
unzip demo.zip
cd demo
./gradlew bootBuildImage

# Tag and upload to a registry
docker tag demo:0.0.1-SNAPSHOT someuser/demo:1.0
docker push someuser/demo:1.0

# Run on Kubernetes
kubectl run demo --image someuser/demo:1.0

# Observe the state
kubectl get pod demo
kubectl logs demo

The state of the pod will be "Error" or "CrashLoopBackoff"

The output of kubectl logs is:

Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate factory class [org.springframework.boot.autoconfigure.BackgroundPreinitializer] for factory type [org.springframework.context.ApplicationListener]
        at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$throwing$0(SpringFactoriesLoader.java:650)
        at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$handleMessage$3(SpringFactoriesLoader.java:674)
        at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:231)
        at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:206)
        at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:160)
        at org.springframework.boot.SpringApplication.getSpringFactoriesInstances(SpringApplication.java:459)
        at org.springframework.boot.SpringApplication.getSpringFactoriesInstances(SpringApplication.java:455)
        at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:274)
        at org.springframework.boot.SpringApplication.<init>(SpringApplication.java:252)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1302)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1291)
        at com.example.demo.DemoApplication.main(DemoApplication.java:10)
Caused by: java.lang.InternalError: java.lang.reflect.InvocationTargetException
        at com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:67)
        at com.oracle.svm.core.containers.Container.metrics(Container.java:44)
        at com.oracle.svm.core.ContainerInfo.<init>(ContainerInfo.java:34)
        at com.oracle.svm.core.Containers.activeProcessorCount(Containers.java:125)
        at java.base@17.0.6/java.lang.Runtime.availableProcessors(Runtime.java:247)
        at org.springframework.boot.autoconfigure.BackgroundPreinitializer.<clinit>(BackgroundPreinitializer.java:68)
        at java.base@17.0.6/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base@17.0.6/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at org.springframework.core.io.support.SpringFactoriesLoader$FactoryInstantiator.instantiate(SpringFactoriesLoader.java:381)
        at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:228)
        ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base@17.0.6/java.lang.reflect.Method.invoke(Method.java:568)
        at com.oracle.svm.core.containers.Metrics.systemMetrics(Metrics.java:63)
        ... 18 more
Caused by: java.lang.ExceptionInInitializerError
        at com.oracle.svm.core.containers.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:78)
        at com.oracle.svm.core.containers.CgroupMetrics.getInstance(CgroupMetrics.java:164)
        ... 20 more
Caused by: java.lang.NullPointerException
        at java.base@17.0.6/java.util.Objects.requireNonNull(Objects.java:208)
        at java.base@17.0.6/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:263)
        at java.base@17.0.6/java.nio.file.Path.of(Path.java:147)
        at java.base@17.0.6/java.nio.file.Paths.get(Paths.java:69)
        at com.oracle.svm.core.containers.CgroupUtil.lambda$readStringValue$0(CgroupUtil.java:57)
        at java.base@17.0.6/java.security.AccessController.executePrivileged(AccessController.java:144)
        at java.base@17.0.6/java.security.AccessController.doPrivileged(AccessController.java:569)
        at com.oracle.svm.core.containers.CgroupUtil.readStringValue(CgroupUtil.java:59)
        at com.oracle.svm.core.containers.CgroupSubsystemController.getStringValue(CgroupSubsystemController.java:66)
        at com.oracle.svm.core.containers.CgroupSubsystemController.getLongValue(CgroupSubsystemController.java:125)
        at com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getLongValue(CgroupV1Subsystem.java:269)
        at com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.getHierarchical(CgroupV1Subsystem.java:215)
        at com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.setSubSystemControllerPath(CgroupV1Subsystem.java:203)
        at com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.initSubSystem(CgroupV1Subsystem.java:111)
        at com.oracle.svm.core.containers.cgroupv1.CgroupV1Subsystem.<clinit>(CgroupV1Subsystem.java:47)
        ... 22 more

Workaround

A workaround is to incorporate the following into the build.gradle:

tasks.named("bootBuildImage") {
    // Workaround for NPE - see https://github.com/oracle/graal/issues/4757
    environment["BP_NATIVE_IMAGE_BUILD_ARGUMENTS"] = "-H:-UseContainerSupport"
}

Answer 1 · 2023-02-21T15:53:25.000Z

@ndwinton Thanks for the report.

A workaround is to incorporate the following into the build.gradle:

Have you verified that this workaround solves the issue in your case?

there is an available workaround (see below) which could be incorporated into the build process pending resolution of the underlying issue.

The BP_NATIVE_IMAGE_BUILD_ARGUMENT environment variable is used by the Paketo Buildpack for Native Image. Spring Boot would not set this environment variable automatically when building an image, as this would mean the configuration would only be set when the Spring Boot build tools are used to build the Docker image, and not when other CNB platforms like pack, kpack, Tekton, etc are used. If it did make sense to do this, we'd want to do it in a way that benefits all CNB platforms equally. This could be done by writing the configuration option to a native image args file or by making the change directly in the buidpack.

Answer 2 · 2023-02-21T21:56:48.000Z

@scottfrederick - Yes, the workaround does work, at least as far as the images I have built are concerned. I obviously can't guarantee it's a universal fix but the logic to it seems sound.

I agree that a solution that works wider than Spring Boot would be great.

Answer 3 · 2023-02-21T22:19:35.000Z

Marking this for the team to discuss what, if anything, we'd want to do to work around what appears to be a GraalVM or JDK bug.

Answer 4 · 2023-05-08T15:56:49.000Z

We've discussed this as a team and decided that this issue does not warrant Spring Boot implementing a workaround for what appears to be a bug in another project. Please continue to use the configuration shown above until a solution is available.

Answer 5 · 2023-06-19T06:49:01.000Z

We've discussed this as a team and decided that this issue does not warrant Spring Boot implementing a workaround for what appears to be a bug in another project. Please continue to use the configuration shown above until a solution is available.

Hi. Recently I find this issue on my client's k8s. In that time, I had to manully build a image base on jar file. It is really annouying,because it doesn't show up on test k8s cluster. Will this issue be fixed later, or it doesn't appear on higher version of springboot?

Answer 6 · 2023-06-19T07:32:08.000Z

As Scott explained, we have no plans to try to work around the problem. If it's important to you, please raise it with the JDK or GraalVM teams.

Answer 7 · 2023-09-06T08:42:47.000Z

I'm using GraalVM 17.0.8+9.1 and Spring Boot 3 to generate native-maven-plugin (native:compile -Pnative) executable (not docker image) and get into same issue, above mentioned workaround is not relevant as I'm not creating docker image, what can be the workaround for my use case ?

Answer 8 · 2023-09-06T10:49:25.000Z

@pavelorehov if you're looking for some support for working around a GraalVM issue, please use one of GraalVM's support channels.