EarthScope/rover

Database is locked (again)

Opened this issue · 1 comments

An instance of rover retrieve request.txt, attached to this issue, run on Python 3.6 has lead to a critical error sqlite3.OperationalError: data base is locked. The rover retrieve.log shows two stack traces that are leading to the data base is locked error.

 CRITICAL 2019-06-25 18:00:08,691: Traceback (most recent call last):
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/__init__.py", line 116, in main
     with ProcessManager(config):
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/process.py", line 48, in __enter__
     self._check_command(self._command)
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/process.py", line 87, in _check_command
     pid, command = self._current_command_inside_transaction()
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/process.py", line 69, in _current_command_inside_transaction
     self._db.execute('DELETE FROM rover_processes WHERE pid = ?', (pid,))
 sqlite3.OperationalError: database is locked
 CRITICAL 2019-06-25 19:47:33,962: Traceback (most recent call last):
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/__init__.py", line 117, in main
     execute(config.command, config)
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/__init__.py", line 96, in execute
     commands[command][0](config).run(config.args)
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/retrieve.py", line 213, in run
     return self.do_run(args, True, RETRIEVE)
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/retrieve.py", line 152, in do_run
     return self._fetch()
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/retrieve.py", line 180, in _fetch
     n_downloads = self._download_manager.download()
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/manager.py", line 882, in download
     self.step(quiet=False)
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/manager.py", line 854, in step
     self._update_stats()
   File "/.conda/envs/rover/lib/python3.6/site-packages/rover/manager.py", line 906, in _update_stats
     self._db.execute('DELETE FROM rover_download_stats', tuple())
 sqlite3.OperationalError: database is locked

Please note that both database is locked operations originate from a DELETE FROM sql command.
request.txt

It seems like this database locking is associated with multiple open transactions that are all trying to write or read from the database at the same time. The busy timeout has been increased to 120 seconds. The busy timeout allows incoming SQL transactions to sleep while another transaction is interfacing with the database, the maximum amount of time that the transaction can sleep is set in seconds by the integer value above Busy Timeout.

A connection.commit() command also been added after the DELETE FROM statements. Committing the Delete FROM statements to the database should allow the database to unlock and be ready for another transaction commit .

The request.txt, above, has been run through this modified code with no locking database exceptions.