Unable to delete index after node has been removed from cluster.
filthz opened this issue · 1 comments
filthz commented
Elassandra 6.2.3.23
I have removed a node from my cluster:
2020-12-14 07:02:39,628 WARN [GossipStage:1] CassandraDiscovery.java:665 onRemove Removing node ip=/10.4.3.7 node={10.4.3.7}{662fa76b-01a9-4a37-ac46-c72a9e3e0811}{662fa76b-01a9-4a37-ac46-c72a9e3e081$
2020-12-14 07:02:39,722 INFO [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] MasterService.java:245 runTasks node-removed-/10.4.3.7[org.elassandra.discovery.CassandraDiscovery$RoutingTableU$
2020-12-14 07:02:39,723 INFO [elasticsearch[10.4.3.8][clusterApplierService#updateTask][T#1]] ClusterApplierService.java:487 applyChanges removed {{10.4.3.7}{662fa76b-01a9-4a37-ac46-c72a9e3e0811}{66$
2020-12-17 07:02:39,846 ERROR [GossipTasks:1] NoSpamLogger.java:97 log java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178)
at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240)
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
at java.lang.Thread.run(Thread.java:748)
After that im not longer able to delete existing elastic indexes:
2020-12-23 16:13:49,917 WARN [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] CassandraDiscovery.java:1125 publishAsParticipator Ignore and acknowlegde obsolete update metadata=662fa76b-01a9-4a37-ac46-c72a9e3e0811/137
2020-12-23 16:13:49,931 WARN [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] AppliedClusterStateAction.java:95 sendAppliedToNode error sending cluster state commit state [662fa76b-01a9-4a37-ac46-c72a9e3e0811/137] to null
java.lang.NullPointerException: discovery node must not be null
at java.util.Objects.requireNonNull(Objects.java:228)
at org.elasticsearch.transport.TransportService.isLocalNode(TransportService.java:1223)
at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:531)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:510)
at org.elassandra.discovery.AppliedClusterStateAction.sendAppliedToNode(AppliedClusterStateAction.java:78)
at org.elassandra.discovery.CassandraDiscovery.publishAsParticipator(CassandraDiscovery.java:1126)
at org.elassandra.discovery.CassandraDiscovery.publish(CassandraDiscovery.java:960)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:251)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:152)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-12-23 16:14:04,682 WARN [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] CassandraDiscovery.java:1125 publishAsParticipator Ignore and acknowlegde obsolete update metadata=662fa76b-01a9-4a37-ac46-c72a9e3e0811/137
2020-12-23 16:14:04,683 WARN [elasticsearch[10.4.3.8][masterService#updateTask][T#1]] AppliedClusterStateAction.java:95 sendAppliedToNode error sending cluster state commit state [662fa76b-01a9-4a37-ac46-c72a9e3e0811/137] to null
java.lang.NullPointerException: discovery node must not be null
How can I fix the indexes / elastic now?
vroyer commented
Update elassandra to the last version.