Function nodes failed to start.
Milky2018 opened this issue · 1 comments
Milky2018 commented
I setup the cluster by following hydro-cluster docs. The function nodes failed to start. Here are the logs:
[ec2-user@ip-172-31-24-229 hydro-project]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
function-nodes-s7r2k 1/4 CrashLoopBackOff 785 (18s ago) 21h
management-pod 1/1 Running 0 22h
memory-nodes-tvxjn 1/1 Running 1 (22h ago) 22h
monitoring-pod 1/1 Running 0 22h
routing-nodes-wjdch 1/1 Running 1 (22h ago) 22h
scheduler-nodes-hwk8m 1/1 Running 0 22h
[ec2-user@ip-172-31-24-229 hydro-project]$ kubectl logs function-nodes-s7r2k
Defaulted container "function-1" out of: function-1, function-2, function-3, cache-container
eth0: error fetching interface information: Device not found
From https://github.com/hydro-project/anna
* [new branch] master -> origin/master
Switched to a new branch 'brnch'
Branch 'brnch' set up to track remote branch 'master' from 'origin'.
Synchronizing submodule url for 'common'
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/anna
copying anna/__init__.py -> build/lib/anna
copying anna/lattices.py -> build/lib/anna
copying anna/base_client.py -> build/lib/anna
copying anna/client.py -> build/lib/anna
copying anna/zmq_util.py -> build/lib/anna
copying anna/common.py -> build/lib/anna
copying anna/cloudburst_pb2.py -> build/lib/anna
copying anna/causal_pb2.py -> build/lib/anna
copying anna/anna_pb2.py -> build/lib/anna
copying anna/shared_pb2.py -> build/lib/anna
running install_lib
copying build/lib/anna/cloudburst_pb2.py -> /usr/local/lib/python3.6/dist-packages/anna
copying build/lib/anna/causal_pb2.py -> /usr/local/lib/python3.6/dist-packages/anna
copying build/lib/anna/anna_pb2.py -> /usr/local/lib/python3.6/dist-packages/anna
copying build/lib/anna/shared_pb2.py -> /usr/local/lib/python3.6/dist-packages/anna
byte-compiling /usr/local/lib/python3.6/dist-packages/anna/cloudburst_pb2.py to cloudburst_pb2.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/anna/causal_pb2.py to causal_pb2.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/anna/anna_pb2.py to anna_pb2.cpython-36.pyc
byte-compiling /usr/local/lib/python3.6/dist-packages/anna/shared_pb2.py to shared_pb2.cpython-36.pyc
running install_egg_info
running egg_info
creating Anna.egg-info
writing Anna.egg-info/PKG-INFO
writing dependency_links to Anna.egg-info/dependency_links.txt
writing requirements to Anna.egg-info/requires.txt
writing top-level names to Anna.egg-info/top_level.txt
writing manifest file 'Anna.egg-info/SOURCES.txt'
reading manifest file 'Anna.egg-info/SOURCES.txt'
writing manifest file 'Anna.egg-info/SOURCES.txt'
removing '/usr/local/lib/python3.6/dist-packages/Anna-0.1-py3.6.egg-info' (and everything under it)
Copying Anna.egg-info to /usr/local/lib/python3.6/dist-packages/Anna-0.1-py3.6.egg-info
running install_scripts
From https://github.com/hydro-project/cloudburst
* [new branch] aft-support -> origin/aft-support
* [new branch] dependabot/pip/protobuf-3.15.0 -> origin/dependabot/pip/protobuf-3.15.0
* [new branch] dependabot/pip/pyyaml-5.4 -> origin/dependabot/pip/pyyaml-5.4
* [new branch] master -> origin/master
Switched to a new branch 'brnch'
Branch 'brnch' set up to track remote branch 'master' from 'origin'.
Synchronizing submodule url for 'common'
Traceback (most recent call last):
File "cloudburst/server/executor/server.py", line 504, in <module>
int(exec_conf['thread_id']))
File "cloudburst/server/executor/server.py", line 104, in executor
status.ip = ip
TypeError: None has type NoneType, but expected one of: bytes, unicode
Milky2018 commented
Eh, I use the latest kOps and the version of Kubernetes is 1.23. I also have changed the images specified in /home/ec2-user/hydro-project/cluster/hydro/cluster/kops/yaml to latest versions ubuntu-based images. Would that matter?