peak/s5cmd

s5cmd ls --region argument, and implied lists respect the cp argument

jordan-jack-schneider opened this issue · 0 comments

If you're doing a copy from one region to another with --no-clobber there are a lot of failure modes with region resolution. The resolution order of ls is

Region detection
While executing the commands, s5cmd detects the region according to the following order of priority:
--source-region or --destination-region flags of cp command.
AWS_REGION environment variable.
Region section of AWS profile.
Auto detection from bucket region (via HeadBucket API call).
us-east-1 as default region.

So this is actually impossible if the region section of your aws profile is set (which you might want as a default). Without the config, it relied on HeadBucket calls, which can sometimes cause performance problems when making a lot of copies with s5cmd run.

Ideally, s5cmd would take in a --region argument, and s5cmd cp $SRC $DST --destination-region $REGION --no-clobber would pass the --destination-region flag to the internal list needed by --no-clobber.