A Buildkite plugin to restore and save
directories by cache keys. For example, use the checksum of a .resolved
or .lock
file
to restore/save built dependencies between independent builds, not just jobs.
With tarball or rsync, if source folder has changes, this will not fail your build, instead will surpress and continue.
For S3, Instead of sync millions of files, I just tarball before S3 operation then copy to s3. This will reduce both time and cost on AWS.
Plus, In addition to tarball & rsync, we also do not re-create another tarball for same cache key if its already exists.
steps:
- plugins:
- gencer/cache#v2.0.5:
cache_key: "v1-cache-{{ checksum 'Podfile.lock' }}"
paths: [ "Pods/", "Rome/" ]
The cache key is a string, which support a crude template system. Currently checksum
is
the only command supported for now. It can be used as in the example above. In this case
the cache key will be determined by executing a checksum (actually sha1sum
) on the
Podfile.lock
file, prepended with v1-cache-
.
This plugin uses AWS S3 cp to cache the paths into a bucket as defined by environment variables defined in your agent.
export BUILDKITE_PLUGIN_CACHE_S3_BUCKET_NAME="my-unique-s3-bucket-name"
export BUILDKITE_PLUGIN_CACHE_S3_PROFILE="my-s3-profile"
The paths are synced using Amazon S3 into your bucket using a structure of
organization-slug/pipeline-slug/cache_key.tar
, as determined by the Buildkite environment
variables.
You can also use rsync to store your files using the rsync_storage
config parameter.
If this is set it will be used as the destination parameter of a rsync -az
command.
steps:
- plugins:
- gencer/cache#v2.0.5:
rsync_storage: '/tmp/buildkite-cache'
cache_key: "v1-cache-{{ checksum 'Podfile.lock' }}"
paths: [ "Pods/", "Rome/" ]
The paths are synced using rsync_storage/cache_key/path
. This is useful for maintaining a local
cache directory, even though this cache is not shared between servers, it can be reused by different
agents/builds.
You can also use tarballs to store your files using the tarball_storage
config parameter.
If this is set it will be used as the destination parameter of a tar -cf
command.
steps:
- plugins:
- gencer/cache#v2.0.5:
tarball_storage: '/tmp/buildkite-cache'
tarball_keep_max_days: 7 # Optional. Removes tarballs older than 7 days.
cache_key: "v1-cache-{{ checksum 'Podfile.lock' }}"
paths: [ "Pods/", "Rome/" ]
The paths are synced using tarball_storage/cache_key.tar
. This is useful for maintaining a local
cache directory, even though this cache is not shared between servers, it can be reused by different
agents/builds.
Original work by @danthorpe
Copyright (C) 2020 Gencer W. Genç. Licensed as MIT.