sakserv/hadoop-mini-clusters

kafka_2.10 to kafka_2.11

Closed this issue · 4 comments

zqhxuyuan commented

plz upgrade kafka_2.10 to kafka_2.11. or else exception throws .

java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
at kafka.message.MessageSet.<init>(MessageSet.scala:71)
	at kafka.message.ByteBufferMessageSet.<init>(ByteBufferMessageSet.scala:259)
	at kafka.message.MessageSet$.<init>(MessageSet.scala:31)
	at kafka.message.MessageSet$.<clinit>(MessageSet.scala)
	at kafka.server.Defaults$.<init>(KafkaConfig.scala:49)
	at kafka.server.Defaults$.<clinit>(KafkaConfig.scala)
	at kafka.log.Defaults$.<init>(LogConfig.scala:35)
	at kafka.log.Defaults$.<clinit>(LogConfig.scala)
	at kafka.log.LogConfig$.<init>(LogConfig.scala:248)
	at kafka.log.LogConfig$.<clinit>(LogConfig.scala)
	at kafka.server.KafkaConfig$.<init>(KafkaConfig.scala:270)
	at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala)
	at kafka.server.KafkaConfig.fromProps(KafkaConfig.scala)
	at com.github.sakserv.minicluster.impl.KafkaLocalBroker.configure(KafkaLocalBroker.java:223)
	at com.github.sakserv.minicluster.impl.KafkaLocalBroker.start(KafkaLocalBroker.java:164)
	at com.zqh.spark.connectors.kafka.KafkaTestBase.setUp(KafkaTestBase.scala:39)
Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 33 more

If I exclude kafka_2.10, and use kafka_2.11, still exceptions:

java.lang.NoClassDefFoundError: kafka/utils/Time
	......
	at com.github.sakserv.minicluster.impl.KafkaLocalBroker.start(KafkaLocalBroker.java:194)
	at com.zqh.spark.connectors.kafka.KafkaTestBase.setUp(KafkaTestBase.scala:39)

I'm using 0.1.13

jetoile commented

Hello,

Since Hortonworks has overload Kafka's constructor, you have to use the hortonworks' kafka version

Hope this helps.

timvw commented

As jetoile already mentionned, make sure to use the same kafka version as hadoop-mini-clusters...

I can confirm that it works for the following versions:

<hadoop-mini-clusters.version>0.1.11</hadoop-mini-clusters.version>
<kafka.version>0.10.0.1</kafka.version>

<dependency>
            <groupId>com.github.sakserv</groupId>
            <artifactId>hadoop-mini-clusters-kafka</artifactId>
            <version>${hadoop-mini-clusters.version}</version>
            <scope>compile</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.10</artifactId>
                </exclusion>
            </exclusions>
</dependency>

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.11</artifactId>
            <version>${kafka.version}</version>
            <scope>compile</scope>
</dependency>
sakserv commented

@zqhxuyuan - truly sorry for the delay. Free time has been limited lately. Can you help me understand the need here? This project has aligned with HDP versions to date, and I believe kafka-2.10 is still what HDP ships with. Maybe we can consolidate the requirements and needs into issue #48 and come up with a solution for both. Thoughts?

sakserv commented

I'm looking to restructure the project for Hadoop/HDP 3, given the wide range of changes. This should make it easier to support different versions of a particular service. Given this, and the lack of discussion on this topic, I'm going to close this issue. Please reopen if this is still needed.