riparias/gbif-alert

Make sure only one import is performed at the time

Opened this issue · 1 comments

We recently had performance issues that made the import take too long (more than 24hrs for RIPARIAS), therefore the import process is restarted the next day while the first one is still in process, this is something to be avoided.

After a quick investigation, I didn't found any integrated mechanism nor easy solution for that. Possible solutions:

  • create a file / an entry in the database / a Redis key to keep track of the running status (similar to Django maintenance mode)
  • actually use the status of django-maintenance-mode (skip import if already in maintenance mode) => doesn't seems great since we only put in maintenance mode in the middle of the process. Also, the site could be in maintenance mode for other reasons than a running import.
  • Check if using RQ's scheduler feature (in place of cron) could help avoiding the issue.