pinterest/DoctorK

Metrics Collector fails to return brokers on localhost

saherahwal opened this issue · 7 comments

saherahwal commented

Hi

I have been struggling to get the Kafka metrics collector to start on broker. I have Kafka cluster running locally with 3 brokers on localhost (on ports 9092-9094)

This is the command I use to run stored in start.sh file.

java -server \
     -Dlog4j.configurationFile=file:../config/log4j2.xml \
     -cp lib/*:kafkastats-0.2.4.9.jar \
     com.pinterest.doctorkafka.stats.KafkaStatsMain \
     -broker 127.0.0.1:9092 \
     -jmxport 9999 \
     -topic brokerstats \
     -zookeeper 127.0.0.1:2181/cluster1 \
     -uptimeinseconds 3600 \
     -pollingintervalinseconds 60 \
     -ostrichport 2051 \
     -tsdhostport localhost:18126 \
     -kafka_config ~/oss/kafka/config/server1.properties \
     -producer_config ~/oss/kafka/config/producer1.properties \
     -primary_network_ifacename eth0

sudo ./start.sh
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
zkUrl:127.0.0.1:2181/cluster1
secPro:PLAINTEXT
brokerStr:null
Exception in thread "main" java.lang.NullPointerException
        at java.util.Hashtable.put(Hashtable.java:460)
        at com.pinterest.doctorkafka.util.OperatorUtil.createKafkaProducerProperties(OperatorUtil.java:239)
        at com.pinterest.doctorkafka.stats.KafkaAvroPublisher.<init>(KafkaAvroPublisher.java:62)
        at com.pinterest.doctorkafka.stats.KafkaStatsMain.main(KafkaStatsMain.java:135)

The list of brokers is empty. When I use zookeeper shell directly I retrieve 3 active brokers just fine.

bin/zookeeper-shell.sh localhost:2181
ls /brokers/ids
[1, 2, 3]

Can someone help with what I am doing wrong here?

saherahwal commented

NOTE: in above

zkUrl:127.0.0.1:2181/cluster1
secPro:PLAINTEXT
brokerStr:null

print statement I added to code myself to double check my hypothesis that list of brokers returned empty.

ambud commented

Here's the bug in your setup:
bin/zookeeper-shell.sh localhost:2181 vs -zookeeper 127.0.0.1:2181/cluster1
So either Kafka brokers should be using the same znode as kafka stats or the other way around.

saherahwal commented

Hi Ambud
I am not sure I understand. Can you please elaborate? I was trying to showcase that zookeeper shell is able to retrieve the brokers just fine. While the Kafka Metrics Collector is using ZkUtil which is not retrieving anything.

I tried both localhost and 127.0.0.1 if that's what you are referring to. Thanks for your help.

ambud commented

Could you please confirm that the zk path used in kafkastats is the same as what the broker is using? From the output posted in the original question that is not the same hence the NPE.

saherahwal commented

Hi @ambud
I am using docker-compose to create multi-broker cluster. Here is relevant part for broker service

environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'

I don't see the /cluster1 part but without it I get JsonMerge exception indicating my path format is not correct.


sudo java -server      -Dlog4j.configurationFile=file:../config/log4j2.xml      -cp lib/*:kafkastats-0.2.4.9.jar      com.pinterest.doctorkafka.stats.KafkaStatsMain      -broker 127.0.0.1      -jmxport 9999      -topic brokerstats      -zookeeper localhost:2181      -uptimeinseconds 3600      -pollingintervalinseconds 60      -ostrichport 2051
   -tsdhostport localhost:18126      -kafka_config ~/oss/kafka/config/server1.properties      -producer_config ~/oss/kafka/config/producer1.properties      -primary_network_ifacename eth0

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
zkUrl:localhost:2181
secPro:PLAINTEXT
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonMerge
        at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.<clinit>(JacksonAnnotationIntrospector.java:50)
        at com.fasterxml.jackson.databind.ObjectMapper.<clinit>(ObjectMapper.java:291)
        at kafka.utils.Json$.<init>(Json.scala:30)
        at kafka.utils.Json$.<clinit>(Json.scala)
        at kafka.zk.BrokerIdZNode$.decode(ZkData.scala:193)
        at kafka.utils.ZkUtils.parseBrokerJson(ZkUtils.scala:708)
        at kafka.utils.ZkUtils.getBrokerInfo(ZkUtils.scala:871)
        at kafka.utils.ZkUtils.$anonfun$getAllBrokersInCluster$2(ZkUtils.scala:280)
        at kafka.utils.ZkUtils.$anonfun$getAllBrokersInCluster$2$adapted(ZkUtils.scala:280)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.utils.ZkUtils.getAllBrokersInCluster(ZkUtils.scala:280)
        at com.pinterest.doctorkafka.util.OperatorUtil.getBrokers(OperatorUtil.java:220)
        at com.pinterest.doctorkafka.util.OperatorUtil.createKafkaProducerProperties(OperatorUtil.java:237)
        at com.pinterest.doctorkafka.stats.KafkaAvroPublisher.<init>(KafkaAvroPublisher.java:62)
        at com.pinterest.doctorkafka.stats.KafkaStatsMain.main(KafkaStatsMain.java:135)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonMerge
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 21 more
saherahwal commented

@ambud
The commit below

a744e501d5a0908eb5a0e9da5750f437e879ac49

removes all dependencies for jackson. @kabochya might know why

commit a744e50
Author: kabochya kabochya@users.noreply.github.com
Date: Fri Sep 27 15:41:49 2019 -0700

bump dropwizard version to 1.3.15 and remove patch dependencies of jetty and jackson (#187)

ambud commented

We haven't tested Dr.K in operations on compose so I am not sure if it will behave as expected. The two exception posted are pointing to two different issues.

The original issue posted is related to znode paths that should be consistent between Dr. kafka, kafkastats and the kafka cluster i.e. all of them should use the same zookeeper path for detection of the kafka cluster.

The second one points to a dependency issue which we will likely need to get back to you on and might have been auto patched via github bot causing this issue.
Note 0.2.4.9 is a very old version of Dr. K. https://github.com/pinterest/doctorkafka/tree/0.3.x is the latest branch in case you are just getting started.