port conflict with random zookeeper ports
Opened this issue · 5 comments
About 1 out of every 50 test runs on my MacOS dev machine I get a port conflict on the randomly assigned zookeeper ports.
Would it be possible to validate that these ports are open before trying? Or catch and retry with different ports?
Or perhaps since zookeeper is not strictly needed with kafka 3.4 is it possible to use a config that does not use zookeeper?
I am currently doing
kafka = provisionWith(defaultClusterConfig());
kafka.start();
and getting
[INFO] Running com...KafkaMarketDataClientInteg1Test
2023-10-10 13:36:55 [Thread-8] ERROR o.a.c.test.TestingZooKeeperServer - From testing server (random state: false) for instance: InstanceSpec{dataDirectory=/var/folders/46/xk_p6xjx3w93pk8wv1tq_5wm0000gn/T/1696959413472-0, port=55521, electionPort=55522, quorumPort=55523, deleteDataDirectoryOnClose=true, serverId=
4, tickTime=-1, maxClientCnxns=-1, customProperties={}, hostname=127.0.0.1} org.apache.curator.test.InstanceSpec@59c48e5e
java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:555)
at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:81)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:676)
at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:109)
at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:105)
at org.apache.curator.test.TestingZooKeeperMain.internalRunFromConfig(TestingZooKeeperMain.java:248)
at org.apache.curator.test.TestingZooKeeperMain.runFromConfig(TestingZooKeeperMain.java:132)
at org.apache.curator.test.TestingZooKeeperServer$1.run(TestingZooKeeperServer.java:158)
at java.base/java.lang.Thread.run(Thread.java:833)
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 51.028 s <<< FAILURE! - in com...KafkaMarketDataClientInteg1Test
[ERROR] com...KafkaMarketDataClientInteg1Test.basicFlowWithFiltering Time elapsed: 51.025 s <<< ERROR!
java.lang.RuntimeException: Unable to start the embedded Kafka cluster.
at net.mguenther.kafka.junit.EmbeddedKafkaCluster.start(EmbeddedKafkaCluster.java:66)
at com...KafkaMarketDataClientInteg1Test.before(KafkaMarketDataClientInteg1Test.java:56)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Suppressed: java.lang.NullPointerException: Cannot invoke "java.util.Map.values()" because "this.brokers" is null
at net.mguenther.kafka.junit.EmbeddedKafkaCluster.stop(EmbeddedKafkaCluster.java:77)
at com...KafkaMarketDataClientInteg1Test.after(KafkaMarketDataClientInteg1Test.java:81)
... 3 more
Caused by: java.lang.RuntimeException: Unable to start an embedded ZooKeeper instance.
at net.mguenther.kafka.junit.EmbeddedZooKeeper.start(EmbeddedZooKeeper.java:30)
at net.mguenther.kafka.junit.EmbeddedKafkaCluster.start(EmbeddedKafkaCluster.java:45)
... 4 more
Caused by: org.apache.curator.test.FailedServerStartException: Timed out waiting for server startup
at org.apache.curator.test.TestingZooKeeperMain.blockUntilStarted(TestingZooKeeperMain.java:147)
at org.apache.curator.test.TestingZooKeeperServer.start(TestingZooKeeperServer.java:167)
at org.apache.curator.test.TestingServer.<init>(TestingServer.java:117)
at org.apache.curator.test.TestingServer.<init>(TestingServer.java:100)
at org.apache.curator.test.TestingServer.<init>(TestingServer.java:63)
at net.mguenther.kafka.junit.EmbeddedZooKeeper.start(EmbeddedZooKeeper.java:25)
... 5 more
Haven't run into this issue yet. Out of curiosity: What are the specs of your system? Proper port management for the ZooKeeper testing server is a concern of the library that Kafka for JUnit uses. We're getting rid of it, once Kafka ships with a production-ready KRaft implementation that works out-of-the-box.
Kafka for JUnit is -in its current latest - tied to Apache Kafka 3.4.x, which comes with an early-access KRaft implementation. Hence, there's still the necessity for ZK. The earliest version that I'd feel comfortable with is 3.5.x, which ships with a production-ready opt-in for KRaft.
The easiest migration path for this library would be to wait for Apache Kafka 4.0.0, which will drop ZK entirely and enable KRaft out-of-the-box. However, I'll check if an opt-in for the next Kafka for JUnit version - which will target Apache Kafka 3.5.x - is feasible.
It's a MacBookPro circa 2019 running Ventura 13.5.2, 32 GB RAM.
lsof shows open listen ports are currently 29754, 49153-49163, 49172, 59578, 60010, 60012, 60014, 62290, 63342.
zookeeper ports from runs with a BindException are
port=49172, electionPort=49173, quorumPort=49174
port=50718, electionPort=50719, quorumPort=50720
port=50997, electionPort=50998, quorumPort=50999
port=55521, electionPort=55522, quorumPort=55523
port=57104, electionPort=57105, quorumPort=57106
port=61593, electionPort=61594, quorumPort=61595
port=62242, electionPort=62243, quorumPort=62244
port=65064, electionPort=65065, quorumPort=65066
port=65121, electionPort=65122, quorumPort=65123
Most of those ports don't show as currently in use, so I assume there were transient and no longer in use.
I would really appreciate a KRaft config with 3.5 if you can do it. Thank you.
I took a look at it, but it is too much of a hassle to introduce an alternative provisioning mechanism, especially with the prospect of ZooKeeper being removed altogether in the very near future. Kafka for JUnit will remove ZK as soon as Kafka ships without it.
org.testcontainers.containers.KafkaContainer supports KRaft mode (and kafka-native).