Help: Kafka topics are deleted automatically after sudo systemctl stop kafka is done.
mai1x9 opened this issue · 2 comments
Can anyone please help out on fixing the issue related to kafka topics being deleted as soon as kafka service is stopped.
OS: Ubuntu 20.04
Kafka Version: 3.3.1
Steps to reproduce:
Topic: "A" is created.
Now run sudo systemctl stop kafka.
After 1 minute, start kafka service again sudo systemctl start kafka.
Now topic A is deleted for some strange reason and I need to recreate again.
On side note,
I have changed kafka log directory to /var/log.kafka folder from ./tmp folder and also updated zookeeper data dir from ./tmp to /var/log/zookeeper
Error logs,
Topic: Security.
When topic describe is run using kafka-topics.sh, output is,
kafkacluster@ml-server-test-kafka-cluster:/usr/local/kafka-server/kafka_2.12-3.3.1/bin$ kafkacluster@ml-server-test-kafka-cluster:/usr/local/kafka-server/kafka_2.12-3.3.1/bin$ ./kafka-topics.sh --describe --bootstrap-server 10.1.0.6:9092 --topic <TOPICNAME>
-bash: syntax error near unexpected token `newline'
kafkacluster@ml-server-test-kafka-cluster:/usr/local/kafka-server/kafka_2.12-3.3.1/bin$ ./kafka-topics.sh --describe --bootstrap-server 10.1.0.6:9092 --topic security
Topic: security TopicId: RfZ8rFydSz2d9IFt-S968g PartitionCount: 12 ReplicationFactor: 1 Configs: cleanup.policy=delete,retention.ms=6048000000,max.message.bytes=524288
Topic: security Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 3 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 4 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 5 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 6 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 7 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 8 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 9 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 10 Leader: 0 Replicas: 0 Isr: 0
Topic: security Partition: 11 Leader: 0 Replicas: 0 Isr: 0
Now stop the kafka server using sudo systemctl stop kafka. Start kafka service again after a second or 2 using sudo systemctl start kafka.
When kafka-topics.sh is run, the output is,
kafkacluster@ml-server-test-kafka-cluster:/usr/local/kafka-server/kafka_2.12-3.3.1/bin$ kafkacluster@ml-server-test-kafka-cluster:/usr/local/kafka-server/kafka_2.12-3.3.1/bin$ ./kafka-topics.sh --describe --bootstrap-server 10.1.0.6:9092 --topic security
[2022-11-19 15:27:16,360] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/10.1.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-11-19 15:27:16,764] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/10.1.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Topic 'security' does not exist as expected
[2022-11-19 15:27:18,260] ERROR java.lang.IllegalArgumentException: Topic 'security' does not exist as expected
at kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:401)
at kafka.admin.TopicCommand$TopicService.describeTopic(TopicCommand.scala:313)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)
Which shows the topic does not exist. However if i rerun the code after 5 to 10 seconds gap, then it shows topic exists and shown all related data.
Why does this weired behaviour occurs? In reality topic exists and is not deleted, however when the producer or consumer tries to conenct to broker as soon as broker is up, I am ending up with topic does not exist error.
Any reasons for it.
Looks like kafka broker takes some time to startup and sending early commend may return topic does not exist.
However this does not mean topic got deleted. it does exist however responds positively only after broker is fully up.