[DevOps] ReqlDriverError: Connection is closed.
Opened this issue · 0 comments
graphicore commented
Seems like the discovery of e.g. the database at the beginning of the lifetime of a container is a bad idea, if the database changes location afterwards.
Traceback (most recent call last):
File "worker-launcher.py", line 207, in consume
run()
File "/var/python/worker/fontbakery.py", line 354, in run
fonts = _prepare(self._job, self._cache, self._dbOps, None)
File "/var/python/worker/fontbakery.py", line 194, in _prepare
dbOps.update({'preparation_logs': logs})
File "/var/python/worker/fontbakery.py", line 91, in update
return self.q.get(self._docid).update(_doc).run(self.conn)
File "/usr/local/lib/python3.7/dist-packages/rethinkdb/ast.py", line 140, in run
return c._start(self, **global_optargs)
File "/usr/local/lib/python3.7/dist-packages/rethinkdb/net.py", line 678, in _start
self.check_open()
File "/usr/local/lib/python3.7/dist-packages/rethinkdb/net.py", line 652, in check_open
raise ReqlDriverError('Connection is closed.')
rethinkdb.errors.ReqlDriverError: Connection is closed.
The workers seem to catch the error and survive this and hence never reconnect to the database then. Therefore, at least this could lead to a fatal pod failure and kill the container. It will then be restarted, so that the next version can discover the database again.