ksafranski/amet

Find better mechanism for syncing

Closed this issue · 1 comments

Per this comment by @TomFrost - using rsync is going to have scans and syncs running full-throttle which is resource intensive. Need to find a solution which doesn't cause such heavy utilization and more elegantly syncs back to the host-mounted volume.

Current Plan:

  • Slow down rsync ops to 30-60m cycles to prevent heavy runs and CPU spikes.
  • Have sync operations generate a "lockfile" to prevent multiple, concurrent runs, release on complete and check timestamp on existing to handle next-run checks
  • Add a termination trap task on the container to ensure when container is exited it runs a final sync.
  • (Maybe) add --bwlimit to rsync to throttle