googlefonts/fontbakery-dashboard

[DevOps] ReqlDriverError: Connection is closed.

Opened this issue · 0 comments

Seems like the discovery of e.g. the database at the beginning of the lifetime of a container is a bad idea, if the database changes location afterwards.

Traceback (most recent call last):
  File "worker-launcher.py", line 207, in consume
    run()
  File "/var/python/worker/fontbakery.py", line 354, in run
    fonts = _prepare(self._job, self._cache, self._dbOps, None)
  File "/var/python/worker/fontbakery.py", line 194, in _prepare
    dbOps.update({'preparation_logs': logs})
  File "/var/python/worker/fontbakery.py", line 91, in update
    return self.q.get(self._docid).update(_doc).run(self.conn)
  File "/usr/local/lib/python3.7/dist-packages/rethinkdb/ast.py", line 140, in run
    return c._start(self, **global_optargs)
  File "/usr/local/lib/python3.7/dist-packages/rethinkdb/net.py", line 678, in _start
    self.check_open()
  File "/usr/local/lib/python3.7/dist-packages/rethinkdb/net.py", line 652, in check_open
    raise ReqlDriverError('Connection is closed.')
rethinkdb.errors.ReqlDriverError: Connection is closed.

The workers seem to catch the error and survive this and hence never reconnect to the database then. Therefore, at least this could lead to a fatal pod failure and kill the container. It will then be restarted, so that the next version can discover the database again.