Stream backups directly to/from S3/HDFS without wasting disk space during the process.
This tool is a ommand line interface for the smart_open library
pip install pystream
Stream mysqldump directly to S3 without wasting any additional disk space during the dump process
mysqldump | gzip | pystream - s3://backups/mysqldump.gz
Restore MySQL backup directly from S3
pystream s3://backups/mysqldump.gz - | gunzip | mysql
Stream a tarball to S3
tar cz . | pystream - s3://backups/backup.tar.gz
Stream a tarball from S3
pystream s3://backups/backup.tar.gz - | tar xz
S3 cat
pystream s3://bucket/path/to/key -
And the usual s3cmd cp like usage:
pystream s3://bucket/path/to/key /path/on/filesystem
pystream /path/on/filesystem s3://bucket/path/to/key