RockefellerArchiveCenter/fornax

Add logic so only one transfer is processed at a time

Closed this issue · 2 comments

Is your feature request related to a problem? Please describe.

When multiple transfers are processed at a single time Archivematica slows to a crawl.

Describe the solution you'd like

Add logic to the approve transfer service which checks to see if ingests or transfers are still being processed, and does not approve the next transfer if so.

Describe alternatives you've considered

Ask @bonniegee how she's addressed this for digitized content.

For digitized content, I added two while loops to check when a Transfer has completed, and when an Ingest has completed. See lines 58-78

There are still a couple of areas for improvement:

  • I do not account for transfer or ingest statuses that are not "completed" or "failed"
  • The loop assumes there is only one transfer or ingest at a time.

Because I've got "Start Transfer" and "Approve Transfer" in two different services, I've implemented this in two different steps:

  • When calling "Start Transfer" the service first looks to see if there are any existing approved transfers. If there are, it just goes back to sleep.
  • When calling "Approve Transfer" the service checks for the status of the last approved ingest (querying the database by process_status and last_modified) and if that status is PROCESSING it goes back to sleep. This is wrapped in a try/except block since it's possible the last approved will still be a transfer (rather than an ingest)