Release-207 intergration test:kafka systemctl status failed and workaround

Question

Release-207 intergration test:kafka systemctl status failed and workaround

Closed this issue 3 years ago · 3 comments

kafka systemctl status failed:

root@yao-instana-vm-32-instana-db:~# systemctl status kafka
● kafka.service - Kafka Server
     Loaded: loaded (/lib/systemd/system/kafka.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2021-10-16 20:27:08 PDT; 11h ago
   Main PID: 119367 (code=exited, status=127)

Oct 16 20:27:08 yao-instana-vm-32-instana-db.fyre.ibm.com systemd[1]: kafka.service: Scheduled restart job, restart counter is at 5.
Oct 16 20:27:08 yao-instana-vm-32-instana-db.fyre.ibm.com systemd[1]: Stopped Kafka Server.
Oct 16 20:27:08 yao-instana-vm-32-instana-db.fyre.ibm.com systemd[1]: kafka.service: Start request repeated too quickly.
Oct 16 20:27:08 yao-instana-vm-32-instana-db.fyre.ibm.com systemd[1]: kafka.service: Failed with result 'exit-code'.
Oct 16 20:27:08 yao-instana-vm-32-instana-db.fyre.ibm.com systemd[1]: Failed to start Kafka Server.

vi /var/log/syslog
...
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: [2021-10-17 09:11:49,272] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: java.nio.file.NoSuchFileException: /etc/kafka/server.properties
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at java.base/java.nio.file.Files.newInputStream(Files.java:155)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:627)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at kafka.Kafka$.getPropsFromArgs(Kafka.scala:51)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at kafka.Kafka$.main(Kafka.scala:67)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db kafka[127338]: #011at kafka.Kafka.main(Kafka.scala)
Oct 17 09:11:49 yao-instana-vm-32-2-instana-db systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILUR
...

Workaround:

cp /opt/kafka/config/server.properties /etc/kafka/server.properties
systemctl stop kafka
systemctl daemon-reload
systemctl start kafka

root@yao-instana-vm-32-instana-db:/etc/kafka# systemctl status kafka
● kafka.service - Kafka Server
     Loaded: loaded (/lib/systemd/system/kafka.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kafka.service.d
             └─override.conf
     Active: active (running) since Sun 2021-10-17 20:30:21 PDT; 3s ago
   Main PID: 213379 (java)
      Tasks: 25 (limit: 38431)
     Memory: 92.2M
     CGroup: /system.slice/kafka.service
             └─213379 /opt/instana/jre11/bin/java -Xmx256M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt>

Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:99)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:86)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:38)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at kafka.log.LogManager.<init>(LogManager.scala:106)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at kafka.log.LogManager$.apply(LogManager.scala:1212)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at kafka.server.KafkaServer.startup(KafkaServer.scala:290)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at kafka.Kafka$.main(Kafka.scala:82)
Oct 17 20:30:23 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]:         at kafka.Kafka.main(Kafka.scala)
Oct 17 20:30:24 yao-instana-vm-32-instana-db.fyre.ibm.com kafka[213379]: [2021-10-17 20:30:24,401] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)

Answer 1 · 2021-10-20T04:00:46.000Z

cc @BonnieKris

Answer 2 · 2021-10-27T09:19:38.000Z

@yaoyao12138 could you help verify it? Thanks.

Answer 3 · 2021-10-27T10:00:18.000Z

@BonnieKris Of course, could you share me the new kafka deb package for test? Thank you.