miner node crashes in kubernetes 1.8
lazzarello opened this issue · 3 comments
running Kubernetes 1.8 in minikube on macOS. Deployment applied without errors. The miner pod outputs the following logs
[lazzarello@strop kuberneteth (master)]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
geth-boot-node-pod 1/1 Running 0 18m
geth-boot-node-setup-pod-sn2rs 1/1 Running 0 18m
geth-miner-deployment-5596c997fd-g2mtm 0/1 CrashLoopBackOff 8 18m
monitor-deployment-5479654c44-6f4sm 2/2 Running 0 18m
[lazzarello@strop kuberneteth (master)]$ kubectl logs geth-miner-deployment-5596c997fd-g2mtm
+ cat /etc/testnet/bootnode/enode.address
+ /usr/local/bin/geth --mine --netrestrict 172.16.0.0/12 --verbosity 3 --config /etc/geth/miner/gethconfig.toml --bootnodes enode://d3b75b24a4fd718b1358d28ba576b2e98f73a7465326c4504f21cc0d124466a91919de18fb72a634f9108f0eedd5ef943aea5c250b41c974c4a7fb7c159b9968@10.108.83.23:30303
Fatal: /etc/geth/miner/gethconfig.toml, line 54: (p2p.Config.BootstrapNodesV5) invalid URL scheme, want "enode"
Fatal: /etc/geth/miner/gethconfig.toml, line 54: (p2p.Config.BootstrapNodesV5) invalid URL scheme, want "enode"
@lazzarello thanks for reporting. i just reproduced it here with k8s 1.8.7 and i can confirm that. i found out it's a race condition within kubernetes, but only when you initially deploy it seems. meaning before the docker images are pulled, which takes some time. so the easiest fix/workaround is to recreate (delete/apply) the deployment, it worked for me. alternatively you can also delete just the miner pod, it should also work. kubelet will recreate it then
still not sure why the scheduler that recreates the pod doesn't self-heal this issue... keeping the issue open for now
@MaximilianMeister
I was wondering if you can help assist as I'm very new to using kubernetes. I was able to deploy without any errors but seems the miner deployment has the CrashLoopBackOff error. I tried to delete/reapply the deployment but still get the same error. Could i be missing some pre-req's?
NAME READY STATUS RESTARTS AGE
geth-boot-node-pod 1/1 Running 0 17m
geth-boot-node-setup-pod-9nbwp 1/1 Running 0 17m
geth-boot-node-setup-pod-kdwhf 1/1 Running 0 17m
geth-miner-deployment-66fbd7cc5f-2r9qk 0/1 CrashLoopBackOff 6 6m
monitor-deployment-6fd9d64489-vmv29 2/2 Running 0 17m
kubectl logs geth-miner-deployment-66fbd7cc5f-2r9qk
Fatal: /etc/geth/miner/gethconfig.toml, line 7: field 'MinerThreads' is not defined in eth.Config, see https://godoc.org/github.com/ethereum/go-ethereum/eth#Config for available fields
Fatal: /etc/geth/miner/gethconfig.toml, line 7: field 'MinerThreads' is not defined in eth.Config, see https://godoc.org/github.com/ethereum/go-ethereum/eth#Config for available fields
@whizkidwho thanks for the hint, i assume that the config file format has changed in geth. try using an older geth version for now. i'll open an issue and try to fix the file format when i have some spare time