xeroxer¶ ↑

Xeroxer is a utility for copying files around with one handy “copy” that takes a source url and a destination url. The handy thing is that it works with http(s), file, and s3 urls. That’s right, one utility to copy things around between exteranl http addresses, local disks, and s3.

Disclaimer: this is an incomplete project as it stands right now. Only three copies work: file to file, s3 to file, and file to s3. Still to be implemented: file to http(s), http(s) to file, s3 to http(s), http(s) to s3, s3 to s3, http(s) to http(s).

Also, it is currently written in 1.8.7 ease because I have a big project still stuck in that wasteland. Eventually, it will be upgraded to 1.9.3 and then to 2.0 ruby.

install¶ ↑

gem install xeroxer

bundle install

configuration¶ ↑

Xeroxer config values are:

:buffer_size => 4194304, # 4 MB is this a good size? hmm.
:storage => :s3,
:s3_domain => "s3.amazonaws.com",
:s3_protocol => "http",
:s3_credentials =>  { :access_key_id => "replace with your s3 access key",
                      :secret_access_key => "replace with your s3 secret key" }

They can be set as follows:

Xeroxer.config["s3_credentials"] = { :access_key_id => "replace with your s3 access key",
                                     :secret_access_key => "replace with your s3 secret key" }

In theory, you can set config settings in a xeroxer.yml file, but this is not tested yet. Still to do is a rails initializer

usage¶ ↑

Simplest form, use the default credentials to copy files to s3 or back.

Xeroxer.copy("file://path/file.ext", "s3://bucket.s3.amazon.com/path/file.ext"

You can set default the credentials for all copies like so:

Xeroxer::configure({:s3_credentials=> {:access_key_id => your_key, :secret_access_key => your_secret}})

Alternatives:

You can set credentials to use to copy to and from a particular bucket. If you use this bucket with specified credentials in an s3:// url, Xeroxer will use the credentials you have specified for that bucket and not the default one. NOT WORKING YET!

Xeroxer::configure({:bucketname => {:s3_credentials=>
                                   {:access_key_id => your_key, :secret_access_key => your_secret}}})

Or you can pass everything in one go: (Passed in credentials to a Xeroxer.copy will override any other credentials given. )

Xeroxer.copy("file://path/file.ext",
             "s3://bucket.s3.amazon.com/path/path/file.ext",
             {:s3_credentials=> {:access_key_id => your_key, :secret_access_key => your_secret}})

You can also copy files around locally, no credentials required, assumed read/write privileges:

Xeroxer.copy("file://path/file1.ext", "file://path/file2.ext"

Relative paths work in urls even though this is not allowed normally. The must start with a “./” to work:

Xeroxer.copy("file://./relativepath/file1.ext", "file://path/file2.ext"

NOT IMPLEMENTED YET:

Xeroxer.copy("file://path/file1.ext", "http://url/path/file2.ext")
Xeroxer.copy("http://url/path/file1.ext", "file://path/file2.ext")
Xeroxer.copy("s3://bucket.s3.amazon.com/path/file.ext", "http://url/path/file2.ext")
Xeroxer.copy("http://url/path/file2.ext", "s3://bucket.s3.amazon.com/path/file.ext")
Xeroxer.copy("http://url/path/file2.ext", "http://url/path/file2.ext")
Xeroxer.copy("s3://bucket.s3.amazon.com/path/file.ext", "s3://bucket.s3.amazon.com/path/file.ext")

testing¶ ↑

To test xeroxer, you need an s3 account and a key/secret pair. Set the system environment variables “bucket” “s3access” and “s3secret” that has read/write access to the bucket specfied in your account. The default value of “bucket” is xeroxer_test, so if you have that bucket on your account you don’t need to set the “bucket” environment variable.

Once the environment variables are setup and you have set all the permissions on your s3 bucket, you can run a test that copies a small file around between endpoints:

bin/rspec

Contributing to xeroxer¶ ↑

Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet.
Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it.
Fork the project.
Start a feature/bugfix branch.
Commit and push until you are happy with your contribution.
Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.
Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.