Automate upload to / download from S3
joshmoore opened this issue · 0 comments
joshmoore commented
Currently the most manual part of the process is migrating files to and from S3. In the docker case, this happens mostly automatically. On AWS, however, it requires several manual steps which become increasingly time consuming with growing file size.
A background process and/or task manager (prefect?) would be ideal to handle these migrations:
- Create two AWS instances ("A" and "B")
- Generate all files on instance A
- Upload all files to S3 from instance A, measuring the upload speeds
- Download all files from S3 to instance B, measuring the download speeds