s32cn is a bash tools to copy a spcificed amazon s3 bucket or bucket/prefix to China S3, and bassed s3cmd.
The usage is s32cn s3://{bucket}/{prefix}/
In this bash scripts, before use, you need edit it and modify following configuration variables:
_source_cfg=/home/ec2-user/.s3cfg-us
_target_cfg=/home/ec2-user/.s3cfg
_target_bucket="sentinel-s2-l2a-xxin"
_temp_download_root_dir=/home/ec2-user/sentinel-2_data
-
Install s3cmd
sudo yum install python-pip sudo pip install s3cmd
-
Configure s3cmd, include 2 config, one for source, the other for China
- Config Global account.
- Run command following command
s3cmd --configure
- Following the commond, input relate information, complete config
- Rename this config file to another name, e.g.
mv .s3cfg .s3cfg-us
- Please note if you want to download open-data with requester_pays mode, you need modify variable requester_pays = True
- Config China account.
- Run command following command
s3cmd --configure
- Following the commond, input relate information, complete config, Please note China region has different end-point, e.g. China NingXia region:
host_base = s3.cn-northwest-1.amazonaws.com.cn host_bucket = %(bucket)s.s3.cn-northwest-1.amazonaws.com.cn
- Config Global account.
-
Download this scripts from github
git clone https://github.com/xinlaoda/s32cn.git
-
Modify variables in this scripts, for example:
_source_cfg=/home/ec2-user/.s3cfg-us _target_cfg=/home/ec2-user/.s3cfg _target_bucket="sentinel-s2-l2a-xxin" _temp_download_root_dir=/home/ec2-user/sentinel-2_data
- _target_bucket is s3 bucket name in China region, you need create it in advance.
- _temp_download_root_dir is the directory full path name in this host to save downloaded data from source s3, you need keep in eye for data volume size.
-
Execute this scripts to copy Global s3 to China s3, for example:
./s32cn s3://sentinel-s2-l2a/tiles/49/S/BU/2021/2/17/0/
then, it will download from source backet, and then upload this data into China s3 bucket, and keep the same directory structure.