Improve behavior when not all cluster nodes have plugin

Question

Improve behavior when not all cluster nodes have plugin

Closed this issue 3 years ago · 1 comments

The existing implementation assumes that all Openfire cluster nodes have the plugin loaded. This is not necessarily the case, if only because a plugin can be reloaded on a node, causing it to be briefly unavailable.

This issue was originally identified when testing a fix for #155. See #156 (comment)

Exceptions like these can be found in the logs, when this issue occurs:

2021.01.08 13:49:15 DEBUG [pool-6-thread-1]: org.jivesoftware.openfire.container.PluginMonitor - Successfully extracted plugin 'monitoring'.
2021.01.08 13:49:15 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginManager - Loading plugin 'monitoring'...
2021.01.08 13:49:16 INFO [pool-155-thread-1]: org.jivesoftware.database.SchemaManager - Found old database version 6 for monitoring. Upgrading to version 7...
2021.01.08 13:49:16 INFO [pool-155-thread-1]: org.jivesoftware.database.SchemaManager - Database update successful.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.graph_002dviewer_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.graph_002dviewer_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.graph_002dviewer_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.graph_002dviewer_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.conversation_002dviewer_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.conversation_002dviewer_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.conversation_002dviewer_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.conversation_002dviewer_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.conversations_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.conversations_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.conversations_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.conversations_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dconversation_002dparticipants_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dconversation_002dparticipants_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dconversation_002dparticipants_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dconversation_002dparticipants_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002dreporter_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002dreporter_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002dreporter_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002dreporter_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002ddashboard_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002ddashboard_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002ddashboard_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.stats_002ddashboard_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dsearch_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dsearch_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dsearch_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.archive_002dsearch_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'org.jivesoftware.openfire.plugin.monitoring.archiving_002dsettings_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'org.jivesoftware.openfire.plugin.monitoring.archiving_002dsettings_jsp' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'org.jivesoftware.openfire.plugin.monitoring.archiving_002dsettings_jsp' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'org.jivesoftware.openfire.plugin.monitoring.archiving_002dsettings_jsp' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'JerseyWrapper' of plugin 'monitoring'...
2021.01.08 13:49:16 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'JerseyWrapper' of plugin 'monitoring'...
2021.01.08 13:49:18 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'JerseyWrapper' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:18 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'JerseyWrapper' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:18 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'GraphServlet' of plugin 'monitoring'...
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'GraphServlet' of plugin 'monitoring'...
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'GraphServlet' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'GraphServlet' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Loading servlet 'ConversationServlet' of plugin 'monitoring'...
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Initializing servlet 'ConversationServlet' of plugin 'monitoring'...
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Registering servlet 'ConversationServlet' of plugin 'monitoring' URL patterns.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginServlet - Servlet 'ConversationServlet' of plugin 'monitoring' loaded successfully.
2021.01.08 13:49:19 DEBUG [pool-monitoring2]: com.reucon.openfire.plugin.archive.impl.DatabaseUpdateSplitJIDsTask - No need to run database table migration to split full JID values into bare JID and resource-part components: configuration indicates that migration already occurred.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.plugin.MonitoringPlugin - Excluding all public resources from the Authorization-Check filter.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.plugin.MonitoringPlugin - Adding the public web sources to the same context as the one that's providing the BOSH interface.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.plugin.MonitoringPlugin - Ensure the JSP engine is initialized correctly (in order to be able to cope with Tomcat/Jasper precompiled JSPs).
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.plugin.MonitoringPlugin - Registering public web context with the embedded webserver.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.apache.jasper.compiler.JspRuntimeContext - Parent class loader is: [org.jivesoftware.openfire.container.PluginClassLoader@b9442fa]
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.apache.jasper.compiler.JspRuntimeContext - Compilation classpath initialized: /usr/local/openfire/plugins/monitoring/classes/:/usr/local/openfire/plugins/monitoring/database/:/usr/local/openfire/plugins/monitoring/i18n/:/usr/local/openfire/plugins/monitoring/web/:/tmp/jetty-0_0_0_0-7070-classes-_monitoring-any-7081121902843690896/jsp:/usr/local/openfire/plugins/monitoring/classes:/usr/local/openfire/plugins/monitoring/database:/usr/local/openfire/plugins/monitoring/i18n:/usr/local/openfire/plugins/monitoring/web
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.apache.jasper.servlet.JspServlet - Scratch dir for the JSP engine is: [/tmp/jetty-0_0_0_0-7070-classes-_monitoring-any-7081121902843690896/jsp]
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.apache.jasper.servlet.JspServlet - IMPORTANT: Do not modify the generated servlets
2021.01.08 13:49:19 DEBUG [hz.openfire.generic-operation.thread-1]: com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager - [172.60.0.10]:5701 [openfire] [3.12.5] Received heartbeat from Member [172.60.0.20]:5701 - e1d05dd7-9dc0-499c-855b-4a36e327a8c4 (now: 2021-01-08 13:49:19.282, timestamp: 2021-01-08 13:49:19.280)
2021.01.08 13:49:19 DEBUG [archive-monitoring-service-worker-2]: org.jivesoftware.openfire.archive.Archiver - Running with max work queue size 500, max purge interval PT1S, grace period PT0.05S.
2021.01.08 13:49:19 DEBUG [archive-monitoring-service-worker-3]: org.jivesoftware.openfire.archive.Archiver - Running with max work queue size 500, max purge interval PT1S, grace period PT0.05S.
2021.01.08 13:49:19 DEBUG [archive-monitoring-service-worker-4]: org.jivesoftware.openfire.archive.Archiver - Running with max work queue size 500, max purge interval PT1S, grace period PT0.05S.
2021.01.08 13:49:19 DEBUG [pool-monitoring2]: org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory - Executing MultiTask: org.jivesoftware.openfire.reporting.stats.GetStatistics
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - Starting...
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - Last modification date: null
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - Unable to parse 'last modification date' value. Returning EPOCH.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[MUCSEARCH] - Starting...
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[MUCSEARCH] - Last modification date: null
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[MUCSEARCH] - Unable to parse 'last modification date' value. Returning EPOCH.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[MESSAGE] - Starting...
2021.01.08 13:49:19 DEBUG [pool-monitoring4]: org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - Rebuilding the Lucene index...
2021.01.08 13:49:19 DEBUG [pool-monitoring4]: org.jivesoftware.openfire.archive.ArchiveIndexer[CONVERSATION] - Removing old data from directory: /usr/local/openfire/monitoring/search
2021.01.08 13:49:19 DEBUG [pool-monitoring3]: org.jivesoftware.openfire.archive.ArchiveIndexer[MUCSEARCH] - Rebuilding the Lucene index...
2021.01.08 13:49:19 DEBUG [hz.openfire.cached.thread-3]: com.hazelcast.internal.partition.InternalPartitionService - [172.60.0.10]:5701 [openfire] [3.12.5] Checking partition state, version: 678
2021.01.08 13:49:19 DEBUG [pool-monitoring3]: org.jivesoftware.openfire.archive.ArchiveIndexer[MUCSEARCH] - Removing old data from directory: /usr/local/openfire/monitoring/mucsearch
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[MESSAGE] - Last modification date: null
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.archive.ArchiveIndexer[MESSAGE] - Unable to parse 'last modification date' value. Returning EPOCH.
2021.01.08 13:49:19 DEBUG [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginManager - Initialized plugin 'monitoring'.
2021.01.08 13:49:19 DEBUG [pool-monitoring6]: org.jivesoftware.openfire.archive.ArchiveIndexer[MESSAGE] - Rebuilding the Lucene index...
2021.01.08 13:49:19 DEBUG [pool-monitoring6]: org.jivesoftware.openfire.archive.ArchiveIndexer[MESSAGE] - Removing old data from directory: /usr/local/openfire/monitoring/msgsearch
2021.01.08 13:49:19 DEBUG [hz.openfire.InvocationMonitorThread]: com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor - [172.60.0.10]:5701 [openfire] [3.12.5] Invocations:1 timeouts:0 backup-timeouts:0
2021.01.08 13:49:19 INFO [pool-155-thread-1]: org.jivesoftware.openfire.container.PluginManager - Successfully loaded plugin 'monitoring'.
2021.01.08 13:49:19 INFO [pool-6-thread-1]: org.jivesoftware.openfire.container.PluginMonitor - Finished processing all plugins.
2021.01.08 13:49:19 ERROR [pool-monitoring2]: org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory - Failed to execute cluster task
java.util.concurrent.ExecutionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: org.jivesoftware.openfire.reporting.stats.GetStatistics
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:92) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:191) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.util.executor.DelegatingFuture.get(DelegatingFuture.java:88) ~[hazelcast-3.12.5.jar!/:?]
at org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory.doSynchronousClusterTask(ClusteredCacheFactory.java:427) [hazelcast-2.5.0.jar!/:?]
at org.jivesoftware.util.cache.CacheFactory.doSynchronousClusterTask(CacheFactory.java:722) [xmppserver-4.6.1.jar:4.6.1]
at org.jivesoftware.openfire.reporting.stats.StatsEngine$SampleTask.run(StatsEngine.java:327) [monitoring-2.2.1-SNAPSHOT.jar!/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: org.jivesoftware.openfire.reporting.stats.GetStatistics
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:88) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.executor.impl.operations.AbstractCallableTaskOperation$OffloadImpl.loadCallable(AbstractCallableTaskOperation.java:106) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.executor.impl.operations.AbstractCallableTaskOperation$OffloadImpl.start(AbstractCallableTaskOperation.java:98) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:224) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:199) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:416) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:153) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:123) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:110) ~[hazelcast-3.12.5.jar!/:?]
at ------ submitted from ------.(Unknown Source) ~[?:?]
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:126) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79) ~[hazelcast-3.12.5.jar!/:?]
... 10 more
Caused by: java.lang.ClassNotFoundException: org.jivesoftware.openfire.reporting.stats.GetStatistics
at org.jivesoftware.openfire.plugin.util.cache.ClusterClassLoader.lambda$loadClass$2(ClusterClassLoader.java:94) ~[hazelcast-2.5.0.jar!/:?]
at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_275]
at org.jivesoftware.openfire.plugin.util.cache.ClusterClassLoader.loadClass(ClusterClassLoader.java:94) ~[hazelcast-2.5.0.jar!/:?]
at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:252) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:699) ~[hazelcast-3.12.5.jar!/:?]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1986) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2160) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) ~[?:1.8.0_275]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) ~[?:1.8.0_275]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) ~[?:1.8.0_275]
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:84) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:77) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.executor.impl.operations.AbstractCallableTaskOperation$OffloadImpl.loadCallable(AbstractCallableTaskOperation.java:106) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.executor.impl.operations.AbstractCallableTaskOperation$OffloadImpl.start(AbstractCallableTaskOperation.java:98) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:224) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:199) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:416) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:153) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:123) ~[hazelcast-3.12.5.jar!/:?]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:110) ~[hazelcast-3.12.5.jar!/:?]
2021.01.08 13:49:19 DEBUG [pool-monitoring2]: org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory - Executing MultiTask: org.jivesoftware.openfire.session.GetSessionsCountTask
2021.01.08 13:49:20 DEBUG [pool-monitoring2]: org.jivesoftware.openfire.plugin.util.cache.ClusteredCacheFactory - Executing MultiTask: org.jivesoftware.openfire.muc.cluster.GetNumberConnectedUsers

Answer 1 · 2021-04-15T10:35:39.000Z

Retrying tasks after a short time could work for briefly unavailable plugins.
After that, trap the HazelcastSerializationException / ClassNotFoundException and write to the Error log that not all nodes had access to the plugin code, then rethrow the error (Task Execution exception?)
There's a difference between running a task on all nodes and running on a specific node, and these may need to be treated differently.