CachePipe is a simple-to-use caching pipeline. By this we mean:
- *simple-to-use*: It is a simple wrapper around existing commands, and
doesn't require you to learn an extensive system
- *caching*: commands are cached based on the contents of an
explicitly-provided dependency list, and future invocations will
not re-run the command if any of them hav changed
- *pipeline*: CachePipe is originally designed to be used as part of a
long pipeline of many different steps
CachePipe uses git-style SHA-1 hashes of the dependencies and the
command invocation itself to determine when changes have been made.
This improves upon existing caching systems, which make use of
unreliable indicators of command completion such as timestamps or even
file existence. It is implemented as a Perl module, and provides both
a Perl API and a command-line interface.
CachePipe supports a number of options. See below for more detail.
-- EXAMPLE -----------------------------------------------------------
# Put this in your init file
export CACHEPIPE=/path/to/cachepipe
export PERL5LIB+=:$CACHEPIPE
. $CACHEPIPE/bashrc
You can now do the following:
- From the command line:
$ cachecmd copy-file "cp a b" a b
[copy-file] rebuilding...
dep=a [CHANGED]
dep=b [NOT FOUND]
cmd=cp a b
took 0 seconds (0s)
$ cachecmd copy-file "cp a b" a b
[copy-file] cached, skipping...
- From a Perl script:
use CachePipe;
my $pipe = new CachePipe();
$pipe->cmd("copy-file","cp a b",["a","b"]);
-- OPTIONS -----------------------------------------------------------
You can tell the command to build the cache files without actually
running the command using the --cache-only flag between the command
name and the command, i.e.
$ cachecmd copy-file --cache-only "cp a b" a b
This will compute the hash over all the dependencies and the command
as if the current state were the desired state. Note that all the
dependencies must exist.
If you have already run a command, you can easily run it again:
$ cachecmd copy-file --rerun
Sometimes you wan to just recompute the hashes of a run, as if it had
just been successfully completed, but without actually re-running the
command. If the command has already been run once, you can type:
$ cachecmd copy-file --cache-only
This flag also works with a new command, e.g.,
$ cachecmd copy-file --cache-only "cp a b" a b
-- AUTHORS -----------------------------------------------------------
This was all Adam Lopez' idea.
Matt Post <post@jhu.edu>
Adam Lopez <alopez@cs.jhu.edu>