ome/bioimage-latency-benchmark

Automate upload to / download from S3

joshmoore opened this issue · 0 comments

Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.

A background process and/or task manager (prefect?) would be ideal to handle these migrations:

  • Create two AWS instances ("A" and "B")
  • Generate all files on instance A
  • Upload all files to S3 from instance A, measuring the upload speeds
  • Download all files from S3 to instance B, measuring the download speeds