apache/ignite

Assistance Needed with Master-Slave Configuration and Preload Modes

Kartikk1999 opened this issue · 2 comments

Kartikk1999 commented

I am reaching out regarding a critical bug in my application currently running in production. The setup consists of three servers, where one server (the master node) is connected to the database, and the other two servers (slave nodes) join the cluster to replicate data from the master node.

Recently, our system experienced downtime, during which the master node was down for a few minutes. After the master node came back online, instead of loading the latest data from the database, it joined the cluster and inadvertently copied stale data from the other nodes.

Here are the approaches I’ve tried so far to resolve this issue:

Set the cache rebalance mode to NONE on the master node and ASYNC on the slave nodes.
Attempted dynamic master assignment using:
ignite.services().deployClusterSingleton("DatabaseSyncService", new DatabaseSyncService(vertx, routerService, emiService));
Tried using backups, but this did not help since the service and database connectivity were lost when the master node went down.
Despite these efforts, the issue persists. I would greatly appreciate any guidance or suggestions to resolve this problem.

shishkovilja commented

@Kartikk1999 , generally speaking, Ignite does not have master nodes, because all data is distributed uniformly. For in-memory cluster rebalance is performed automatically.

What do you mean as 'stale' data? What are your cache and cluster configurations?