SCC: Crash in findSharedClassImpl3
tajila opened this issue · 11 comments
tajila commented
3XMCPUTIME CPU usage total: 0.299532649 secs, current category="Application"
3XMHEAPALLOC Heap bytes allocated since last GC cycle=829536 (0xCA860)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at com/ibm/oti/shared/SharedClassURLHelperImpl.findSharedClassImpl3(Native Method)
4XESTACKTRACE at com/ibm/oti/shared/SharedClassURLHelperImpl.findSharedClass(SharedClassURLHelperImpl.java:147(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/cds/CDSBundleFile.getClassBytes(CDSBundleFile.java:155)
4XESTACKTRACE at org/eclipse/osgi/internal/cds/CDSBundleFile.getEntry(CDSBundleFile.java:88)
4XESTACKTRACE at org/eclipse/osgi/storage/bundlefile/BundleFileWrapper.getEntry(BundleFileWrapper.java:55(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/classpath/ClasspathEntry.findEntry(ClasspathEntry.java:195(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/classpath/ClasspathManager.findClassImpl(ClasspathManager.java:621(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/classpath/ClasspathManager.findLocalClassImpl(ClasspathManager.java:607(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/classpath/ClasspathManager.findLocalClassImpl(ClasspathManager.java:587(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/classpath/ClasspathManager.findLocalClass(ClasspathManager.java:566(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/ModuleClassLoader.findLocalClass(ModuleClassLoader.java:335(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/BundleLoader.findLocalClass(BundleLoader.java:397(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/sources/PackageSource.getSourceFromLoader(PackageSource.java:180(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/loader/sources/PackageSource.isServiceAssignableTo(PackageSource.java:135)
4XESTACKTRACE at org/eclipse/osgi/internal/serviceregistry/ServiceRegistrationImpl.isAssignableTo(ServiceRegistrationImpl.java:737(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/serviceregistry/ServiceRegistry.isAssignableTo(ServiceRegistry.java:1246(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/serviceregistry/ServiceRegistry.getServiceReferences(ServiceRegistry.java:399(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/framework/BundleContextImpl.getServiceReferences(BundleContextImpl.java:568)
4XESTACKTRACE at org/osgi/util/tracker/ServiceTracker.getInitialReferences(ServiceTracker.java:342)
4XESTACKTRACE at org/osgi/util/tracker/ServiceTracker.open(ServiceTracker.java:303)
5XESTACKTRACE (entered lock: org/osgi/util/tracker/ServiceTracker$Tracked@0x00000000FF7C70F8, entry count: 1)
5XESTACKTRACE (entered lock: org/osgi/util/tracker/ServiceTracker@0x00000000FF7C6E20, entry count: 1)
4XESTACKTRACE at org/osgi/util/tracker/ServiceTracker.open(ServiceTracker.java:267)
4XESTACKTRACE at org/apache/felix/scr/impl/helper/ConfigAdminTracker.<init>(ConfigAdminTracker.java:86)
4XESTACKTRACE at org/apache/felix/scr/impl/BundleComponentActivator.<init>(BundleComponentActivator.java:224)
4XESTACKTRACE at org/apache/felix/scr/impl/Activator.loadComponents(Activator.java:612)
4XESTACKTRACE at org/apache/felix/scr/impl/Activator.access$200(Activator.java:75)
4XESTACKTRACE at org/apache/felix/scr/impl/Activator$ScrExtension.start(Activator.java:480)
4XESTACKTRACE at org/apache/felix/scr/impl/AbstractExtender.createExtension(AbstractExtender.java:196)
4XESTACKTRACE at org/apache/felix/scr/impl/AbstractExtender.modifiedBundle(AbstractExtender.java:169)
4XESTACKTRACE at org/apache/felix/scr/impl/AbstractExtender.modifiedBundle(AbstractExtender.java:49)
4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.customizerModified(BundleTracker.java:490)
4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.customizerModified(BundleTracker.java:1)
4XESTACKTRACE at org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java:234)
4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(BundleTracker.java:452)
4XESTACKTRACE at org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEvent(BundleContextImpl.java:949(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(EventManager.java:234(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:229(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:138)
4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:130(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:217(Compiled Code))
4XESTACKTRACE at org/eclipse/osgi/container/Module.publishEvent(Module.java:499)
4XESTACKTRACE at org/eclipse/osgi/container/Module.start(Module.java:486)
4XESTACKTRACE at org/eclipse/osgi/container/ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1852)
4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128(Compiled Code))
4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
4XESTACKTRACE at java/lang/Thread.run(Thread.java:839)
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK protectedBacktrace+0x12 (0x00007F883AF92DF2 [libj9prt29.so+0x25df2])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK omrintrospect_backtrace_thread_raw+0xbe (0x00007F883AF932CE [libj9prt29.so+0x262ce])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK omrintrospect_backtrace_thread+0x87 (0x00007F883AF92C77 [libj9prt29.so+0x25c77])
4XENATIVESTACK setup_native_thread+0x1e3 (0x00007F883AF93C43 [libj9prt29.so+0x26c43])
4XENATIVESTACK omrintrospect_threads_startDo_with_signal+0x41f (0x00007F883AF94D9F [libj9prt29.so+0x27d9f])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK _ZN18JavaCoreDumpWriter28writeThreadsWithNativeStacksEv+0x430 (0x00007F883AB20F60 [libj9dmp29.so+0x19f60])
4XENATIVESTACK protectedWriteThreadsWithNativeStacks+0xd (0x00007F883AB2180D [libj9dmp29.so+0x1a80d])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK _ZN18JavaCoreDumpWriter18writeThreadSectionEv+0x14b (0x00007F883AB1DBFB [libj9dmp29.so+0x16bfb])
4XENATIVESTACK protectedWriteSection+0x1d (0x00007F883AB18A1D [libj9dmp29.so+0x11a1d])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK _ZN18JavaCoreDumpWriterC2EPKcP16J9RASdumpContextP14J9RASdumpAgent+0x3f5 (0x00007F883AB19F05 [libj9dmp29.so+0x12f05])
4XENATIVESTACK runJavadump+0x1c (0x00007F883AB2414C [libj9dmp29.so+0x1d14c])
4XENATIVESTACK doJavaDump+0x42 (0x00007F883AB0C192 [libj9dmp29.so+0x5192])
4XENATIVESTACK protectedDumpFunction+0x15 (0x00007F883AB0B7D5 [libj9dmp29.so+0x47d5])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK runDumpFunction+0x62 (0x00007F883AB0EFC2 [libj9dmp29.so+0x7fc2])
4XENATIVESTACK runDumpAgent+0x15d (0x00007F883AB0F14D [libj9dmp29.so+0x814d])
4XENATIVESTACK triggerDumpAgents+0x615 (0x00007F883AB26965 [libj9dmp29.so+0x1f965])
4XENATIVESTACK generateDiagnosticFiles+0x162 (0x00007F883B0404E2 [libj9vm29.so+0x424e2])
4XENATIVESTACK omrsig_protect+0x239 (0x00007F883AF973C9 [libj9prt29.so+0x2a3c9])
4XENATIVESTACK vmSignalHandler+0x17a (0x00007F883B04066A [libj9vm29.so+0x4266a])
4XENATIVESTACK mainSynchSignalHandler+0x238 (0x00007F883AF96978 [libj9prt29.so+0x29978])
4XENATIVESTACK (0x00007F883B694520 [libc.so.6+0x42520])
4XENATIVESTACK pool_newElement+0x7e (0x00007F883992CC3E [libjclse29.so+0x5ac3e])
4XENATIVESTACK hashTableAddNodeInList+0x9c (0x00007F883992A58C [libjclse29.so+0x5858c])
4XENATIVESTACK getCachedString+0x142 (0x00007F883990B4E2 [libjclse29.so+0x394e2])
4XENATIVESTACK getCpeTypeForProtocol+0x186 (0x00007F883990BA56 [libjclse29.so+0x39a56])
4XENATIVESTACK Java_com_ibm_oti_shared_SharedClassURLHelperImpl_findSharedClassImpl3+0x317 (0x00007F883990D617 [libjclse29.so+0x3b617])
4XENATIVESTACK (0x00007F87BE5585D5 [<unknown>+0x0])
NULL
tajila commented
hangshao0 commented
@theresa-m Could you look at this ?
hangshao0 commented
hangshao0 commented
#19769 removed the call of getCpeTypeForProtocol() outside the jclCacheMutex, if that is the cause of this crash, the issue should go away in the latest build.
tjwatson commented
Is OpenLiberty/open-liberty/issues/29486 issue similar to this issue and perhaps is already fixed?
tajila commented
Is OpenLiberty/open-liberty#29486 issue similar to this issue and perhaps is already fixed?
Looks familiar, Ill double check
tajila commented
@tjwatson Can you add a description of what your test was doing? We have tests that run a liberty checkpoint/restore scenario in containers but never saw this issue. Perhaps we need more iterations?
tjwatson commented
@tjwatson Can you add a description of what your test was doing? We have tests that run a liberty checkpoint/restore scenario in containers but never saw this issue. Perhaps we need more iterations?
I don't think this had anything to do with our InstantOn testing. Rather it seems like it could be the fact that we have a single liberty installation on the machine running the tests and that installation gets a shared classes cache established that is shared across multiple servers that are run during the FATs. We have many runs against this same cache, so it may be the fact that we have a much larger number of runs against the single cache?
Our other test infrastructure has not been updated to the latest release of Liberty so I am not confident if this happens more broadly or not for other test buckets in Liberty (non-InstantOn).
tajila commented
Is OpenLiberty/open-liberty#29486 issue similar to this issue and perhaps is already fixed?
This issue has been fixed and will be in the 3Q update