AutoMQ/automq

[BUG] Partition reassignment deadlock

Closed this issue · 0 comments

Currently, the node shares a common ElasticReplicaManager#partitionOpExecutor (default 256) thread pool for opening and closing. When there is a large-scale reassignment of partitions, a partition reassignment deadlock may occur.
Let's take an example:

  1. Suppose the thread pool size of partitionOpExecutor is 1. In the cluster, there are two nodes, node1 and node2, with partitions P1 and P2 on node1, and partitions P3 and P4 on node2.
  2. If we reassign P1 and P2 from node1 to node2, and P3 and P4 from node2 to node1,
  3. It is possible that node1 will try to open P3 and node2 will try to open P1, occupying the threads in partitionOpExecutor.
  4. Opening P1 depends on node1 closing P1. Due to step 3 occupying the threads in partitionOpExecutor, it becomes unable to close.
  5. Deadlock occurs.