clusterfuzz service crashing on fuzzbots
nguyendon opened this issue · 2 comments
nguyendon commented
I'm running into an issue on our deploy of v2.6.0 where the fuzz bots and the clusterfuzz service are created but immediately crash.
Apr 29 09:12:16 clusterfuzz-linux-21mc bash[1463]: + sudo -E -H -u clusterfuzz bash -c 'PATH='\'':/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-15-openjdk-amd64/bin'\'' PYTHONPATH='\''/mnt/scratch0/clusterfuzz/src'\'' LD_L
IBRARY_PATH='\'''\'' python3.7 /mnt/scratch0/clusterfuzz/src/python/bot/startup/run.py'
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: Traceback (most recent call last):
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/python/bot/startup/run.py", line 29, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from clusterfuzz._internal.base.untrusted import untrusted_noop
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/base/untrusted.py", line 18, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from clusterfuzz._internal.base import tasks
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/base/tasks.py", line 22, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from clusterfuzz._internal.base import external_tasks
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/base/external_tasks.py", line 16, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from clusterfuzz._internal.datastore import data_handler
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/datastore/data_handler.py", line 30, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from google.cloud import ndb
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/__init__.py", line 28, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from google.cloud.ndb.client import Client
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 18, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: import grpc
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: File "/mnt/scratch0/clusterfuzz/src/third_party/grpc/__init__.py", line 23, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: from grpc._cython import cygrpc as _cygrpc
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/mnt/scratch0/clusterfuzz/src/third_party/grpc/_cython/__init__.py)
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Main process exited, code=exited, status=1/FAILURE
Apr 29 09:12:18 clusterfuzz-linux-21mc docker[1908]: Error response from daemon: No such container: clusterfuzz
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Control process exited, code=exited, status=1/FAILURE
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Failed with result 'exit-code'.
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Scheduled restart job, restart counter is at 2.
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: Stopped clusterfuzz.service.
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: Starting clusterfuzz.service...
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/mnt/scratch0/clusterfuzz/src/third_party/grpc/_cython/__init__.py)
root@clusterfuzz-linux-21mc:/mnt/scratch0# python3.7 -m pip list
Package Version
----------------------- ---------
absl-py 1.0.0
astunparse 1.6.3
cachetools 5.0.0
certifi 2021.10.8
charset-normalizer 2.0.11
crcmod 1.7
distlib 0.3.4
filelock 3.4.2
future 0.17.1
gast 0.3.3
google-auth 2.6.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
grpcio 1.43.0
h5py 2.10.0
idna 3.3
importlib-metadata 4.10.1
Keras-Preprocessing 1.1.2
Markdown 3.3.6
numpy 1.18.5
oauthlib 3.2.0
opt-einsum 3.3.0
pip 19.2.3
pipenv 2022.1.8
platformdirs 2.4.1
protobuf 3.15.3
psutil 5.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
requests 2.27.1
requests-oauthlib 1.3.1
rsa 4.8
scipy 1.4.1
setuptools 41.2.0
six 1.12.0
tensorboard 2.8.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow 2.3.0
tensorflow-estimator 2.3.0
termcolor 1.1.0
typing-extensions 4.0.1
urllib3 1.26.8
virtualenv 20.13.1
virtualenv-clone 0.5.7
Werkzeug 2.0.2
wheel 0.37.1
wrapt 1.13.3
zipp 3.7.0
The fuzz bot images seem to be pinned to gcr.io/clusterfuzz-images/base:a2f4dd6-202202070654. Has anyone else seen this issue?
nguyendon commented
Actually I think I know what my issue is. I patched v2.6.0 to use python38 since python37 wasnt working. The deploy bundle is packaged with python38 so that is probably why its failing on the fuzzbot image.
nguyendon commented
Yup that was the issue.