Always be downloading
balshor opened this issue · 1 comments
balshor commented
When downloading multiple files (s3cp -r), s3cp performs a checksum after each file before moving on to the next. It would be more efficient to start the download of the next file in parallel with checksum on the previous file as the checksum is (presumably) CPU bound while the download is IO bound. Instead of immediately retrying, files that need to be re-downloaded due to corruption can be added to the end of the download queue for retrying later.
aboisvert commented
Catchy title! I like it.