s3-multiputter.py
Description:
Reads a huge file (or device) and uploads to Amazon S3.
Multiple workers are launched which read & send in parallel.
Workers are allocated one chunk of the file at a time.
Usage:
s3-multiputter.py <BUCKET> <FILE> <THREADS> <CHUNKSIZE>
BUCKET: The S3 bucket name to upload to
FILE: The source file to upload
THREADS: Number of parallel uploader threads
CHUNKSIZE: Size (MB) of each chunk
This script was designed for uploading very large block
device images to S3. The problem it aims to solve is moving
an EBS volume from one AWS region to another using S3 as the
transport medium.
Launch a hefty EC2 instance like a c1.xlarge, attach your EBS
volume(s) to it, and use s3-multiputter.py to upload the block
device to S3 -- it can read directly from /dev/sdxN devices.
You can upload directly from an EC2 instance to the S3 service
in another region, or you can upload from an EC2 instance to
the local S3 service and use S3 copy to move your image file
from one S3 region to another.
Prerequisites:
Boto library must be installed & configured with AWS creds
http://code.google.com/p/boto/wiki/BotoConfig