JVM Crash during C2 Compilation

Question

JVM Crash during C2 Compilation

Opened this issue 2 years ago · 17 comments

Describe the bug

During the C2 compilation application crashes. It happens only a couple of times in a day on more than 100 instances

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=18002, tid=18040
#
# JRE version: OpenJDK Runtime Environment Corretto-17.0.6.10.1 (17.0.6+10) (build 17.0.6+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-17.0.6.10.1 (17.0.6+10-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  0x0000000000000000
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/server/core.18002)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-17/issues/
#

---------------  S U M M A R Y ------------

Host: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz, 8 cores, 15G, Ubuntu 18.04.6 LTS
Time: Tue Feb  7 11:56:29 2023 UTC elapsed time: 575.014781 seconds (0d 0h 9m 35s)

---------------  T H R E A D  ---------------

Current thread (0x00007f796c102630):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=18040, stack(0x00007f7931bfc000,0x00007f7931cfd000)]


Current CompileTask:
C2: 575014 68221   !   4       kotlinx.coroutines.flow.AbstractFlow::collect (189 bytes)

To Reproduce

Steps and (source) code to reproduce the behavior.
I don't have any method to reproduce the issue; it always happens during the C2 compiler phase.

Expected behavior

A clear and concise description of what you expected to happen.
JVM shouldn't crash during C2 compiler

Platform information

OS: Ubuntu 18.04.6 LTS
JDK version 17.0.6+10-LTS

hs_err_pid18002.log
replay_pid18002.log

Answer 1 · 2023-02-10T13:19:30.000Z

Thanks for reporting and sharing the log files. If you see this often, can you please upload a few more hs_err_pid log files so we can look for commonalities?

Answer 2 · 2023-02-10T13:23:22.000Z

thanks for the quick reply @olivergillespie. It happens a couple of times a day and when it happens it always shows Kotlin coroutines. I will provide more hs_err_pid today or next week when it happens

Answer 3 · 2023-02-10T16:52:29.000Z

Seems related to #57. We're still looking for ways to reproduce the issue.

Answer 4 · 2023-02-14T07:56:45.000Z

@olivergillespie you can download new crash reports from the links (1,2) I will remove the links from ticket once you have them.

ps: I have also created AWS ticket, and the case id is 11986608411

Answer 5 · 2023-02-14T10:55:10.000Z

Thanks, downloaded.

We're still looking for ways to reproduce the issue.

Just to set expectations, the crash logs will provide us useful clues but this is unlikely to be resolved quickly without a standalone reproducer we can use to debug and fix the issue. We might be able to create one with what we know so far, but if anyone has a on this we'll be glad to make use of it.

Answer 6 · 2023-02-14T12:06:27.000Z

Thanks for the investigation @olivergillespie. Unfortunately, I don't have a way of reproducing issues. It happens randomly and has always happened on the kotlin code compilation.

Answer 7 · 2023-02-15T00:43:41.000Z

hi, @umutkocasarac
Are those 2 hyperlinks correct? I attempt to download them but they both looping back.
I hope you could teach us how to reproduce it. Did you save the coredump file?

Answer 8 · 2023-02-16T21:19:03.000Z

@navyxliu I have restored the links in my previous comment and you can find another crash report here (1,2). I will delete the links in a couple of days (please let me know once you downloaded them).

Unfortunately, JVM is not creating core dump files during the crash, even though it says it will be stored working folder.

Answer 9 · 2023-02-21T19:02:24.000Z

I understand. I got those files.

HotSpot attempts to create a coredump file by default unless you explicitly disable it using -XX:-CreateCoredumpOnCrash.

It's up to your system. There are different circumstances to block it. On Linux, you may try "ulimit -c unlimited"

Answer 10 · 2023-03-03T15:44:07.000Z

hi @navyxliu One of my colleagues reproduced the issue on the simple project. You can download the project from here link

Let me know if you face any issues while running the project

Answer 11 · 2023-03-09T20:44:12.000Z

@umutkocasarac I tried to run your reproducer.
mvn clean package spring-boot:repackage failed:

[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.example:grpc-contracts-java:jar:1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.springframework.boot:spring-boot-maven-plugin is missing. @ org.example:grpc-contracts:1.0-SNAPSHOT, /local/home/eastig/tmp/11/grpc-test/grpc-contracts/pom.xml, line 100, column 21
[WARNING] 'dependencyManagement.dependencies.dependency.exclusions.exclusion.artifactId' for org.quartz-scheduler:quartz:jar with value '*' does not match a valid id pattern. @ org.springframework.boot:spring-boot-dependencies:2.5.14, /home/eastig/.m2/repository/org/springframework/boot/spring-boot-dependencies/2.5.14/spring-boot-dependencies-2.5.14.pom, line 1811, column 25
...
[WARNING]
Mar 09, 2023 6:01:32 PM org.sonatype.guice.bean.reflect.Logs$JULSink warn
WARNING: Error injecting: kr.motd.maven.os.DetectExtension
com.google.inject.ConfigurationException: Guice configuration errors:

1) Could not find a suitable constructor in kr.motd.maven.os.DetectExtension. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at kr.motd.maven.os.DetectExtension.class(Unknown Source)
  while locating kr.motd.maven.os.DetectExtension

1 error
        at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1023)
        at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:982)
        at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1032)
        at org.sonatype.guice.bean.reflect.AbstractDeferredClass.get(AbstractDeferredClass.java:45)
        at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)
        at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:70)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:100)
        at org.sonatype.guice.plexus.lifecycles.PlexusLifecycleManager.onProvision(PlexusLifecycleManager.java:138)
        at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
        at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:55)
        at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:68)
        at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1054)
        at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
        at com.google.inject.Scopes$1$1.get(Scopes.java:59)
        at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
        at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:997)
        at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1047)
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:993)
        at org.sonatype.guice.bean.locators.LazyBeanEntry.getValue(LazyBeanEntry.java:83)
        at org.sonatype.guice.plexus.locators.LazyPlexusBean.getValue(LazyPlexusBean.java:49)
        at org.sonatype.guice.bean.locators.EntryListAdapter$ValueIterator.next(EntryListAdapter.java:112)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
        at org.apache.maven.DefaultMaven.getLifecycleParticipants(DefaultMaven.java:538)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:270)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:414)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:357)
[ERROR] Internal error: com.google.inject.ProvisionException: Guice provision errors:
[ERROR]
[ERROR] 1) Could not find a suitable constructor in kr.motd.maven.os.DetectExtension. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
[ERROR] at kr.motd.maven.os.DetectExtension.class(Unknown Source)
[ERROR] while locating kr.motd.maven.os.DetectExtension
[ERROR] at ClassRealm[extension>kr.motd.maven:os-maven-plugin:1.6.2, parent: sun.misc.Launcher$AppClassLoader@70dea4e]
[ERROR] at ClassRealm[extension>kr.motd.maven:os-maven-plugin:1.6.2, parent: sun.misc.Launcher$AppClassLoader@70dea4e]
[ERROR] while locating org.apache.maven.AbstractMavenLifecycleParticipant annotated with @com.google.inject.name.Named(value=detect-os)
[ERROR]
[ERROR] 1 error
[ERROR] -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: com.google.inject.ProvisionException: Guice provision errors:

1) Could not find a suitable constructor in kr.motd.maven.os.DetectExtension. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at kr.motd.maven.os.DetectExtension.class(Unknown Source)
  while locating kr.motd.maven.os.DetectExtension
  at ClassRealm[extension>kr.motd.maven:os-maven-plugin:1.6.2, parent: sun.misc.Launcher$AppClassLoader@70dea4e]
  at ClassRealm[extension>kr.motd.maven:os-maven-plugin:1.6.2, parent: sun.misc.Launcher$AppClassLoader@70dea4e]
  while locating org.apache.maven.AbstractMavenLifecycleParticipant annotated with @com.google.inject.name.Named(value=detect-os)

1 error
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:168)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:414)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:357)
Caused by: com.google.inject.ProvisionException: Guice provision errors:

1) Could not find a suitable constructor in kr.motd.maven.os.DetectExtension. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at kr.motd.maven.os.DetectExtension.class(Unknown Source)
  while locating kr.motd.maven.os.DetectExtension
  at ClassRealm[extension>kr.motd.maven:os-maven-plugin:1.6.2, parent: sun.misc.Launcher$AppClassLoader@70dea4e]
  at ClassRealm[extension>kr.motd.maven:os-maven-plugin:1.6.2, parent: sun.misc.Launcher$AppClassLoader@70dea4e]
  while locating org.apache.maven.AbstractMavenLifecycleParticipant annotated with @com.google.inject.name.Named(value=detect-os)

1 error
        at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1006)
        at org.sonatype.guice.bean.locators.LazyBeanEntry.getValue(LazyBeanEntry.java:83)
        at org.sonatype.guice.plexus.locators.LazyPlexusBean.getValue(LazyPlexusBean.java:49)
        at org.sonatype.guice.bean.locators.EntryListAdapter$ValueIterator.next(EntryListAdapter.java:112)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
        at org.apache.maven.DefaultMaven.getLifecycleParticipants(DefaultMaven.java:538)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:270)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        ... 11 more
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/InternalErrorException

Do you know how to fix this?

Answer 12 · 2023-03-09T20:49:11.000Z

@umutkocasarac I fixed the issue by using maven 3.9.0.

Answer 13 · 2023-03-09T22:50:07.000Z

@umutkocasarac Thank you for the reproducer. I managed to get a crash.

Answer 14 · 2023-03-13T08:44:36.000Z

Last week we started to exclude some of the Kotlin coroutines classes and grpc server class from the C2 compiler as a workaround and we are not seeing any jvm crashes since then.

Let us know if there is anything we can do for a permanent fix for the issue

-XX:CompileCommand="exclude,kotlinx.coroutines.flow.AbstractFlow::collect"
-XX:CompileCommand="exclude,kotlinx.coroutines.flow.SafeFlow::collectSafely"
-XX:CompileCommand="exclude,io.grpc.kotlin.ServerCalls*::*"

Answer 15 · 2023-03-14T18:26:15.000Z

I tried the project on both macos and linux.
I run "mvn clean package spring-boot:repackage" and it succeeded.
how to trigger the crash?

Answer 16 · 2023-03-14T19:12:50.000Z

I tried the project on both macos and linux. I run "mvn clean package spring-boot:repackage" and it succeeded. how to trigger the crash?

You need to run:

docker build -f docker/Dockerfile  -t jvm-crash .
docker run -it jvm-crash

Answer 17 · 2023-03-14T19:14:05.000Z

This is from README.md. It worked for me.