geneontology/biolink-api

Floating invocation variables can prevent GO API from reinstantiating itself

Closed this issue · 2 comments

kltm commented

Today, @dustine32 and @sierra-moxon recorded an instance here the GO API endpoint restarting failed when the python code was invoked

[2022-04-22 19:38:53 +0000] [1105] [INFO] Starting gunicorn 19.9.0
[2022-04-22 19:38:53 +0000] [1105] [INFO] Listening at: http://0.0.0.0:8888 (1105)
[2022-04-22 19:38:53 +0000] [1105] [INFO] Using worker: gevent
[2022-04-22 19:38:53 +0000] [1108] [INFO] Booting worker with pid: 1108
[2022-04-22 19:38:55 +0000] [1108] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 203, in init_process
    super(GeventWorker, self).init_process()
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/home/ubuntu/biolink-api/wsgi.py", line 1, in <module>
    from biolink.app import app
  File "/home/ubuntu/biolink-api/biolink/__init__.py", line 5, in <module>
    VERSION=git.Repo(search_parent_directories=True).head.object.hexsha[:7]
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/refs/symbolic.py", line 193, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/objects/base.py", line 64, in new_from_sha
    oinfo = repo.odb.info(sha1)
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/db.py", line 37, in info
    hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/cmd.py", line 1077, in get_object_header
    return self.__get_object_header(cmd, ref)
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/cmd.py", line 1066, in __get_object_header
    return self._parse_object_header(cmd.stdout.readline())
  File "/home/ubuntu/biolink-api/venv/lib/python3.6/site-packages/git/cmd.py", line 1028, in _parse_object_header
    raise ValueError("SHA could not be resolved, git returned: %r" % (header_line.strip()))
ValueError: SHA could not be resolved, git returned: b''

The key in this case was

VERSION=git.Repo(search_parent_directories=True).head.object.hexsha[:7]

@sierra-moxon fixed this in place on the production instance. We'd have to redo this if the instance was terminated or we switched instances, but we should be fine for now for restarts.

This is an interesting class of problems where versions changes to things that are not pinned can affect us on restarts.

kltm commented

@sierra-moxon @dustine32 , feel free to close after adding any more detail that would be publicly useful.

biolink.log on the physical machine held the error that pointed us to the solution.