Can not create and delete an index after setting up multidatacenter elassandra cluster
yosiasm opened this issue · 2 comments
Elassandra version: elassandra-6.8.4.11
Plugins installed: []
JVM version (java -version):openjdk version "1.8.0_275", OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_275-b01), OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.275-b01, mixed mode)
OS version (uname -a if on a Unix-like system): Linux TJ-ELASSANDRA-01 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
when I create or deleting an index the request time to elasticsearch api takes forever and nothing changed.
curl -XDELETE http://localhost:9200/entity_aliasing/
curl -XPUT http://localhost:9200/alias_service/
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
- setup 3 new elassandra node as DC2
- change seed config in 3 existing elassandra node (DC1)
- add DC2 in some keyspace replication setting
- nodetool repair
- nodetool rebuild
- then cannot create or delete index from elasticsearch anymore
Please provide the following information:
-
elassandra logs (logs/system.logs or /var/lib/cassandra/system.log)
2021-03-17 05:54:38,029 TRACE [elasticsearch[10.10.10.201][scheduler][T#1]] IndexingMemoryController.java:328 runUnlocked total indexing heap bytes used [0b] vs indices.memory.index_buffer_size [387.3mb], currently writing bytes [0b] -
elasticsearch cluster state (curl http://localhost:9200/_cluster/state)
cluster_state.txt -
cassandra schema (cqlsh>DESC KEYSPACE <your_keyspace>)
describe_keyspace.txt -
cassandra gossip state (run: nodetool gossipinfo)
gossipinfo.txt
I found out that elastic_admin is replicated only on DC2, the solution is to replicate in DC1 and DC2