Zookeeper cluster issue if ZOO_MY_ID not specified

Question

Zookeeper cluster issue if ZOO_MY_ID not specified

amrit-35 opened this issue 4 years ago · 1 comments

Expected behavior

If ZOO_MY_ID not provided as ENV variable. It should automatically generate a different ID for each zookeeper Server.

Actual behavior

Currently if the ID is not provided it will set it as 1 no matter the number of zookeeper servers. For example, if we have 3 servers and we don't specify the id, it will set ZOO_MY_ID as 1 for all the servers as specified in docker-entrypoint.sh
The error in the logs is not really helpful in this particular case.

SERVER1

zoo1_1  | 2021-02-07 00:45:51,689 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-2:QuorumCnxManager@400] - Cannot open channel to 2 at election address zoo2/172.27.0.4:3888
zoo1_1  | java.net.ConnectException: Connection refused (Connection refused)
zoo1_1  | 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
zoo1_1  | 	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
zoo1_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
zoo1_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
zoo1_1  | 	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
zoo1_1  | 	at java.base/java.net.Socket.connect(Unknown Source)
zoo1_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
zoo1_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
zoo1_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
zoo1_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
zoo1_1  | 	at java.base/java.lang.Thread.run(Unknown Source)
zoo1_1  | 2021-02-07 00:45:51,689 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-3:QuorumCnxManager@400] - Cannot open channel to 3 at election address zoo3/172.27.0.3:3888
zoo1_1  | java.net.ConnectException: Connection refused (Connection refused)
zoo1_1  | 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
zoo1_1  | 	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
zoo1_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
zoo1_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
zoo1_1  | 	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
zoo1_1  | 	at java.base/java.net.Socket.connect(Unknown Source)
zoo1_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
zoo1_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
zoo1_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
zoo1_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
zoo1_1  | 	at java.base/java.lang.Thread.run(Unknown Source)

SERVER2

zoo2_1  | 2021-02-07 00:45:55,389 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-2:QuorumCnxManager@400] -Cannot open channel to 3 at election address zoo3/172.27.0.3:3888
zoo2_1  | java.net.ConnectException: Connection refused (Connection refused)
zoo2_1  | 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
zoo2_1  | 	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
zoo2_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
zoo2_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
zoo2_1  | 	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
zoo2_1  | 	at java.base/java.net.Socket.connect(Unknown Source)
zoo2_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
zoo2_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
zoo2_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
zoo2_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
zoo2_1  | 	at java.base/java.lang.Thread.run(Unknown Source)
zoo2_1  | 2021-02-07 00:45:55,389 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-3:QuorumCnxManager@400] - Cannot open channel to 2 at election address /0.0.0.0:3888
zoo2_1  | java.net.ConnectException: Connection refused (Connection refused)
zoo2_1  | 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
zoo2_1  | 	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
zoo2_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
zoo2_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
zoo2_1  | 	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
zoo2_1  | 	at java.base/java.net.Socket.connect(Unknown Source)
zoo2_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
zoo2_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
zoo2_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
zoo2_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
zoo2_1  | 	at java.base/java.lang.Thread.run(Unknown Source)

SERVER3

zoo3_1  | 2021-02-07 00:45:55,320 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-3:QuorumCnxManager@400] - Cannot open channel to 3 at election address /0.0.0.0:3888
zoo3_1  | java.net.ConnectException: Connection refused (Connection refused)
zoo3_1  | 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
zoo3_1  | 	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
zoo3_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
zoo3_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
zoo3_1  | 	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
zoo3_1  | 	at java.base/java.net.Socket.connect(Unknown Source)
zoo3_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
zoo3_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
zoo3_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
zoo3_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
zoo3_1  | 	at java.base/java.lang.Thread.run(Unknown Source)
zoo3_1  | 2021-02-07 00:45:55,320 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-2:QuorumCnxManager@400] - Cannot open channel to 2 at election address zoo2/172.27.0.4:3888
zoo3_1  | java.net.ConnectException: Connection refused (Connection refused)
zoo3_1  | 	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
zoo3_1  | 	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
zoo3_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
zoo3_1  | 	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
zoo3_1  | 	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
zoo3_1  | 	at java.base/java.net.Socket.connect(Unknown Source)
zoo3_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383)
zoo3_1  | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
zoo3_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
zoo3_1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
zoo3_1  | 	at java.base/java.lang.Thread.run(Unknown Source)

Steps to reproduce the behavior

version: '3.1'

services:
  zoo1:
    image: zookeeper
    restart: always
    hostname: zoo1
    ports:
      - 2181:2181
    environment:
      # ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo2:
    image: zookeeper
    restart: always
    hostname: zoo2
    ports:
      - 2182:2181
    environment:
      # ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181

  zoo3:
    image: zookeeper
    restart: always
    hostname: zoo3
    ports:
      - 2183:2181
    environment:
      # ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181

Answer 1 · 2021-02-21T07:36:10.000Z

The suggested behaviour of auto generating IDs for Zookeeper cluster instances is not feasible to implement. Because, not only ZOO_MY_ID must be set, but also ZOO_SERVERS must contain a list of all servers in a cluster along with their IDs. However, a starting Zookeeper container wouldn't know what IDs were generated for other containers in the cluster.