A Buildkite plugin to store ephemeral cache files between builds.
Often builds involve fetching and processing large amounts of data that don't change much between builds, for instance downloading npm/gem/pip/cocoapod packages from central registries, or shared compile cache for things like ccache, or large virtual machine images that can be re-used.
Buildkite recommends using Artifacts for build artifacts that are the result of a build and useful for humans, where as we see cache as being an optional byproduct of builds that doesn't need to be content addressable.
For example, caching the node_modules
folder as long as the package-lock.json
file does not change can be done as follows:
steps:
- label: ':nodejs: Install dependencies'
command: npm ci
plugins:
- cache#v1.0.0:
manifest: package-lock.json
path: node_modules
restore: file
save: file
The file or folder to cache.
The maximum caching level to restore, if available. See the available caching levels
The level to use for saving the cache. See the available caching levels
Defines how the cache is stored and restored. Can be any string (see Customizable Backends), but the plugin natively supports the following:
fs
(default)s3
Very basic local filesystem backend.
The BUILDKITE_PLUGIN_FS_CACHE_FOLDER
environment variable defines where the copies are (default: /var/cache/buildkite
). If you don't change it, you will need to make sure that the folder exists and buildkite-agent
has the propper permissions, otherwise the plugin will fail.
IMPORTANT: the fs
backend just copies files to a different location in the current agent, as it is not a shared or external resource, its caching possibilities are quite limited.
Store things in an S3 bucket. You need to make sure that the aws
command is available and appropriately configured.
You also need the agent to have access to the following defined environment variables:
BUILDKITE_PLUGIN_S3_CACHE_BUCKET
: the bucket to use (backend will fail if not defined)BUILDKITE_PLUGIN_S3_CACHE_PREFIX
: optional prefix to use for the cache within the bucketBUILDKITE_PLUGIN_S3_CACHE_ENDPOINT
: optional S3 custom endpoint to use
Setting the BUILDKITE_PLUGIN_S3_CACHE_ONLY_SHOW_ERRORS
environment variable will reduce logging of file operations towards S3.
Allows for the cached file/folder to be saved/restored as a single file. You will need to make sure to use the same compression when saving and restoring or it will cause a cache miss.
Assuming the underlying executables are available, the allowed values are:
tgz
:tar
with gzip compressionzip
:(un)zip
compression
A path to a file or folder that will be hashed to create file-level caches.
It will cause an unrecoverable error if either save
or restore
are set to file
and this option is not specified.
This plugin uses the following hierarchical structure for caches to be valid (meaning usable), from the most specific to the more general:
file
: only as long as a manifest file does not change (see themanifest
option)step
: valid only for the current stepbranch
: when the pipeline executes in the context of the current branchpipeline
: all builds and steps of the pipelineall
: all the time
When restoring from cache, all levels, in the described order, up to the one specified will be checked. The first one available will be restored and no further levels or checks will be made.
One of the greatest flexibilities of this plugin is its flexible backend architecture. You can provide whatever value you want for the backend
option of this plugin (X
for example) as long as there is an executable script accessible to the agent named cache_X
that respects the following execution protocol:
cache_X exists $KEY
Should exit successfully (0 return code) if any previous call to this very same plugin was made with cache_x save $KEY
. Any other exit code will mean that there is no valid cache and will be ignored.
cache_X get $KEY $FILENAME
Will restore whatever was previously saved on $KEY
(using the save
call described next) to the file or folder $FILENAME
. A non-0 exit code will cause the whole execution to halth and the current step to fail.
You can assume that all calls like this will be preceded by an exists
call to ensure that there is something to get.
cache_X save $KEY $FILENAME
Will save whatever is in the $FILENAME
path (which can be a file or folder) in a way that can be identified by the string $KEY
. A non-0 return code will cause the whole execution to halt and the current step to fail.
- should fail with error 255 on any instance, preferably without output
You can always have more complicated logic by using the plugin multiple times with different levels and on different steps. In the following example the node_modules
folder will be saved and restored with the following logic:
- first step:
- if the
package-lock.json
file has not changed,node_modules
will be restored as is, run thenpm install
(that should do nothing because no dependencies changed), and skip saving the cache because it already exists - if the
package-lock.json
file has changed, it will restore step-level, branch-level and pipeline-level caches of thenode_modules
folder (the first one that exists), runnpm install
(that should be quick, just installing the differences), and then save the resultingnode_modules
folder as a file-level cache
- if the
- second step:
- will restore the file-level cache of the
node_modules
folder saved by the first step and runnpm test
- will restore the file-level cache of the
- third step (that will only run on the
master
branch):- will restore the file-level cache saved by the first step, run
npm run deploy
and finally save the contents of thenode_modules
folder as a pipeline-level cache for usage as a basis even when the lockfile changes (in the first step)
- will restore the file-level cache saved by the first step, run
steps:
- label: ':nodejs: Install dependencies'
command: npm ci
plugins:
- cache#v1.0.0:
manifest: package-lock.json
path: node_modules
restore: pipeline
save: file
- wait: ~
- label: ':test_tube: Run tests'
command: npm test # does not save cache, not necessary
plugins:
- cache#v1.0.0:
manifest: package-lock.json
path: node_modules
restore: file
- wait: ~ # don't run deploy until tests pass
- label: ':goal_net: Save stable cache after deployment'
if: build.branch == "master"
command: npm run deploy
plugins:
- cache#v1.0.0:
manifest: package-lock.json
path: node_modules
restore: file
save: pipeline
MIT (see LICENSE)