Issue starting up cleanly in Kubernetes
Opened this issue · 0 comments
zsiegel commented
Hello
I have been running your great images in kubernetes for quite some time now without issue.
I recently upgraded my GKE cluster to the latest version 1.7.6
I rebooted and redeployed fresh images and zookeeper seems to be unable to start up cleanly.
New Monit id: 91d371eb870b43197094a6ee3df98d75
Stored in '/root/.monit.id'
Starting Monit 5.20.0 daemon with http interface at [*]:2812
'zookeeper-2596185815-p7b71' Monit 5.20.0 started
'zk-service' process is not running
'zk-service' trying to restart
'zk-service' start: '/opt/zk/bin/zk-service.sh start'
2017-10-07 05:39:51,413 [myid:] - INFO [main:QuorumPeerConfig@124] - Reading configuration from: /opt/zk/bin/../conf/zoo.cfg
2017-10-07 05:39:51,428 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2017-10-07 05:39:51,428 [myid:] - ERROR [main:QuorumPeerConfig@301] - Invalid configuration, only one server specified (ignoring)
2017-10-07 05:39:51,430 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2017-10-07 05:39:51,430 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1
2017-10-07 05:39:51,431 [myid:] - WARN [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running in standalone mode
2017-10-07 05:39:51,433 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2017-10-07 05:39:51,448 [myid:] - INFO [main:QuorumPeerConfig@124] - Reading configuration from: /opt/zk/bin/../conf/zoo.cfg
2017-10-07 05:39:51,449 [myid:] - INFO [main:QuorumPeer$QuorumServer@149] - Resolved hostname: 127.0.0.1 to address: /127.0.0.1
2017-10-07 05:39:51,449 [myid:] - ERROR [main:QuorumPeerConfig@301] - Invalid configuration, only one server specified (ignoring)
2017-10-07 05:39:51,449 [myid:] - INFO [main:ZooKeeperServerMain@96] - Starting server
2017-10-07 05:39:51,461 [myid:] - INFO [main:Environment@100] - Server environment:zookeeper.version=3.4.9-1757313, built on 08/23/2016 06:50 GMT
2017-10-07 05:39:51,461 [myid:] - INFO [main:Environment@100] - Server environment:host.name=zookeeper-2596185815-p7b71
2017-10-07 05:39:51,462 [myid:] - INFO [main:Environment@100] - Server environment:java.version=1.8.0_112
2017-10-07 05:39:51,462 [myid:] - INFO [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2017-10-07 05:39:51,462 [myid:] - INFO [main:Environment@100] - Server environment:java.home=/opt/jre1.8.0_112
2017-10-07 05:39:51,467 [myid:] - INFO [main:Environment@100] - Server environment:java.class.path=/opt/zk/bin/../build/classes:/opt/zk/bin/../build/lib/*.jar:/opt/zk/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zk/bin/../lib/slf4j-api-1.6.1.jar:/opt/zk/bin/../lib/netty-3.10.5.Final.jar:/opt/zk/bin/../lib/log4j-1.2.16.jar:/opt/zk/bin/../lib/jline-0.9.94.jar:/opt/zk/bin/../zookeeper-3.4.9.jar:/opt/zk/bin/../src/java/lib/*.jar:/opt/zk/bin/../conf:
2017-10-07 05:39:51,467 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-10-07 05:39:51,467 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
2017-10-07 05:39:51,467 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler=<NA>
2017-10-07 05:39:51,468 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux
2017-10-07 05:39:51,468 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
2017-10-07 05:39:51,468 [myid:] - INFO [main:Environment@100] - Server environment:os.version=4.4.64+
2017-10-07 05:39:51,469 [myid:] - INFO [main:Environment@100] - Server environment:user.name=root
2017-10-07 05:39:51,469 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/root
2017-10-07 05:39:51,469 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/opt/zk
2017-10-07 05:39:51,471 [myid:] - INFO [main:ZooKeeperServer@815] - tickTime set to 2000
2017-10-07 05:39:51,471 [myid:] - INFO [main:ZooKeeperServer@824] - minSessionTimeout set to -1
2017-10-07 05:39:51,471 [myid:] - INFO [main:ZooKeeperServer@833] - maxSessionTimeout set to -1
2017-10-07 05:39:51,475 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2017-10-07 05:39:51,483 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
'zk-service' failed to start (exit status 0) -- '/opt/zk/bin/zk-service.sh start': rm: can't remove '/opt/zk/logs/zookeeper.out': No such file or directory
rm: can't remove '/opt/zk/nohup.out': No such file or directory
ZooKeeper JMX enabled by default
Using config: /opt/zk/bin/../conf/zoo.cfg
I poked around the filesystem and the files it is looking for (zookeeper.out and nohup.out) are indeed there.
Any ideas? When I build and run the same image locally it works just fine.