"i like to move it, move it" - Reel 2 Real
FileUtils for all the other file systems in your life: scp, ssh, s3, hdfs
The idea is seamless mv and cp between two hosts/formats, and neither of those machines have to be the box running the code (i.e. will ssh if needed and run commands on a remote machine)
pre-alpha. what is implemented is functional, but it still falls fall short of the overall goal.
under active development.
If you want to run the tests:
- you'll need to be able to
sshintolocalhostusingid_rsa - you'll need
hadoopsetup with the ability to connect to an hdfs server (otherwise the hadoop tests will simply be pending)
(basket)[http://github.com/jashmenn/basket/tree/master]
we have the logfiles in a local folder and we want to put them in hdfs in chunks we have the logfiles in a remote folder and we want to put them in hdfs in chunks we have the logfiles in s3 and we want to put them in a folder in chunks we have the logfiles in s3 and we want to put them in hdfs in chunks
we have the logfiles in hdfs in chunks and we want to run mapred jobs on them we have the results in hdfs in chunks and we want to load them into hive
Copyright (c) 2009 Nate Murray, AT&T Interactive