Change graphite service.port results in pod crashloopbackoff
Closed this issue · 2 comments
Is this a request for help?:
yes
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Version of Helm and Kubernetes:
helm: v3.1.2
k8s: v1.15.5
Which chart in which version:
graphite: 0.5.0
What happened:
I changed the service.port from the default 8080 to port 8000 and the pod gets stuck in a crashloopbackoff with the liveness and readiness probes failing to connect.
What you expected to happen:
I expected the service.port to be configurable to a different port number.
How to reproduce it (as minimally and precisely as possible):
I ran the following on my mac terminal connected with docker desktop and k8s through docker desktop.
helm install graphite --set service.port=8000 --set service.type=NodePort kiwigrid/graphite
Anything else we need to know:
Everything works as expected with the service.type=NodePort if I don't change the service.port.
Below are the logs when i follow the pod after changing the service.port:
$ kubectl logs graphite-0 --follow
ok: run: nginx: (pid 21) 0s
22 Apr 17:07:27 - [40] reading config file: /opt/statsd/config/udp.js
22 Apr 17:07:27 - server is up INFO
Starting carbon-aggregator (instance a)
22/04/2020 17:07:27 :: [console] Using sorted write strategy for cache
22/04/2020 17:07:27 :: [aggregator] reading new aggregation rules from /opt/graphite/conf/aggregation-rules.conf
22/04/2020 17:07:27 :: [aggregator] clearing aggregation buffers
22/04/2020 17:07:27 :: [clients] connecting to carbon daemon at 127.0.0.1:2004:None
22/04/2020 17:07:27 :: [console] twistd 18.9.0 (/opt/graphite/bin/python3 3.6.6) starting up.
22/04/2020 17:07:27 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
22/04/2020 17:07:27 :: [console] Installing SIG_IGN for SIGHUP
22/04/2020 17:07:27 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2004:None)
22/04/2020 17:07:27 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::startedConnecting (127.0.0.1:2004)
22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2023
22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7fabdbf33c18>
22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2024
22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7fabdbf33c88>
22/04/2020 17:07:27 :: [console] <twisted.internet.tcp.Connector object at 0x7fabdbf33630> will retry in 2 seconds
22/04/2020 17:07:27 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::clientConnectionFailed (127.0.0.1:2004) Connection was refused by other side: 111: Connection refused.
22/04/2020 17:07:27 :: [clients] Destination is down: 127.0.0.1:2004:None (1/5)
22/04/2020 17:07:27 :: [console] Stopping factory CarbonClientFactory(127.0.0.1:2004:None)
Starting carbon-cache (instance a)
22/04/2020 17:07:27 :: [console] Using sorted write strategy for cache
22/04/2020 17:07:27 :: [console] Enabling Whisper fallocate support
22/04/2020 17:07:27 :: [console] twistd 18.9.0 (/opt/graphite/bin/python3 3.6.6) starting up.
22/04/2020 17:07:27 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
22/04/2020 17:07:27 :: [console] ServerFactory starting on 7002
22/04/2020 17:07:27 :: [console] Starting factory <twisted.internet.protocol.ServerFactory object at 0x7f2a891cc940>
22/04/2020 17:07:27 :: [console] Installing SIG_IGN for SIGHUP
22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2003
22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7f2a88efc470>
22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2004
22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7f2a88efc4e0>
22/04/2020 17:07:30 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2004:None)
22/04/2020 17:07:30 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::startedConnecting (127.0.0.1:2004)
22/04/2020 17:07:30 :: [clients] CarbonClientProtocol(127.0.0.1:2004:None)::connectionMade
22/04/2020 17:07:30 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::connectionMade (CarbonClientProtocol(127.0.0.1:2004:None))
22/04/2020 17:07:30 :: [clients] Destination is up: 127.0.0.1:2004:None
22/04/2020 17:07:30 :: [listener] MetricPickleReceiver connection with 127.0.0.1:37928 established
22/04/2020 17:07:37 :: [tagdb] Tagging carbon.agents.graphite-0-a.memUsage
22/04/2020 17:07:38 :: [listener] MetricLineReceiver connection with 127.0.0.1:41142 established
22/04/2020 17:07:38 :: [listener] MetricLineReceiver connection with 127.0.0.1:41142 closed cleanly
22/04/2020 17:07:38 :: [tagdb] Tagged carbon.agents.graphite-0-a.memUsage in 0.09088778495788574
22/04/2020 17:07:48 :: [listener] MetricLineReceiver connection with 127.0.0.1:41204 established
22/04/2020 17:07:48 :: [listener] MetricLineReceiver connection with 127.0.0.1:41204 closed cleanly
The value only changes the kubernetes config, not the graphite config. So you have to change the GRAPHITE_URL variable in the graphite config too.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.