Floating invocation variables can prevent GO API from reinstantiating itself
Closed this issue · 2 comments
kltm commented
Today, @dustine32 and @sierra-moxon recorded an instance here the GO API endpoint restarting failed when the python code was invoked
[2022-04-22 19:38:53 +0000] [1105] [INFO] Starting gunicorn 19.9.0
[2022-04-22 19:38:53 +0000] [1105] [INFO] Listening at: http://0.0.0.0:8888 (1105)
[2022-04-22 19:38:53 +0000] [1105] [INFO] Using worker: gevent
[2022-04-22 19:38:53 +0000] [1108] [INFO] Booting worker with pid: 1108
[2022-04-22 19:38:55 +0000] [1108] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 203, in init_process
super(GeventWorker, self).init_process()
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
__import__(module)
File "/home/ubuntu/biolink-api/wsgi.py", line 1, in <module>
from biolink.app import app
File "/home/ubuntu/biolink-api/biolink/__init__.py", line 5, in <module>
VERSION=git.Repo(search_parent_directories=True).head.object.hexsha[:7]
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/refs/symbolic.py", line 193, in _get_object
return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/objects/base.py", line 64, in new_from_sha
oinfo = repo.odb.info(sha1)
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/db.py", line 37, in info
hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/cmd.py", line 1077, in get_object_header
return self.__get_object_header(cmd, ref)
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/cmd.py", line 1066, in __get_object_header
return self._parse_object_header(cmd.stdout.readline())
File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/cmd.py", line 1028, in _parse_object_header
raise ValueError("SHA could not be resolved, git returned: %r" % (header_line.strip()))
ValueError: SHA could not be resolved, git returned: b''
The key in this case was
VERSION=git.Repo(search_parent_directories=True).head.object.hexsha[:7]
@sierra-moxon fixed this in place on the production instance. We'd have to redo this if the instance was terminated or we switched instances, but we should be fine for now for restarts.
This is an interesting class of problems where versions changes to things that are not pinned can affect us on restarts.
kltm commented
@sierra-moxon @dustine32 , feel free to close after adding any more detail that would be publicly useful.
sierra-moxon commented
biolink.log on the physical machine held the error that pointed us to the solution.