Buildpipe
A Buildkite plugin to dynamically generate pipelines. Especially useful for monorepos where you want to create dependencies between projects.
Example
initial_pipeline.yml
steps:
- label: ":buildkite:"
plugins:
- jwplayer/buildpipe#v0.9.1:
dynamic_pipeline: dynamic_pipeline.yml
dynamic_pipeline.yml
projects:
- label: project1
path: project1/ # changes in this directory will trigger steps for project1
skip:
- deploy* # skip steps with label matching deploy* (e.g. deploy-prd)
- test
- label: project2
skip: test
path:
- project2/
- project1 # you can trigger a project using multiple paths
- label: project3
skip:
- deploy-stg
path: project3/somedir/ # subpaths can also be triggered
steps: # the same schema as regular buildkite pipeline steps
- label: test
env:
BUILDPIPE_SCOPE: project # this variable ensures a test step is generated for each project
command:
- cd $$BUILDPIPE_PROJECT_PATH # BUILDPIPE_PROJECT_PATH is set by buildpipe
- make test
- wait
- label: build
branches: "master"
env:
BUILDPIPE_SCOPE: project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make build
- make publish-image
agents:
- queue=build
- wait
- label: tag
branches: "master"
command:
- make tag-release
- wait
- label: deploy-stg
branches: "master"
concurrency: 1
concurrency_group: deploy-stg
env:
BUILDPIPE_SCOPE: project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make deploy-staging
- wait
- block: ":rocket: Release!"
branches: "master"
- wait
- label: deploy-prd
branches: "master"
concurrency: 1
concurrency_group: deploy-prd
env:
BUILDPIPE_SCOPE: project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make deploy-prod
The above pipelines specify the following:
- There are three projects to track in the repository.
- The env variable
BUILDPIPE_SCOPE: project
tells buildpipe to generate a step for each project if that project changed. - The
skip
option will skip any step label matchingdeploy*
. - The env variable
BUILDPIPE_PROJECT_PATH
is created by buildpipe as the project's path. If multiple paths are specified for a project, it's the first path.
Full working example
For a full working example, check out Buildkite Monorepo Example.
Configuration
Plugin
Option | Required | Type | Default | Description |
---|---|---|---|---|
default_branch | No | string | master | Default branch of repository |
diff_pr | No | string | Override command for non-default branch (see below for a better explanation of the defaults) | |
diff_default | No | string | Override command for default branch (see below for a better explanation of the defaults) | |
dynamic_pipeline | Yes | string | The name including the path to the pipeline that contains all the actual steps | |
log_level | No | string | INFO | The Level of logging to be used by the python script underneath; pass DEBUG for verbose logging if errors occur |
Project schema
Option | Required | Type | Default | Description |
---|---|---|---|---|
label | Yes | string | Project label | |
path | Yes | array | The path(s) that specify changes to a project | |
skip | No | array | Exclude steps that have labels that match the rule |
Other useful things to note:
- Option
skip
make use of Unix shell-style wildcards (Look at .gitignore files for inspiration) - If multiple paths are specified, the environment variable
BUILDPIPE_PROJECT_PATH
will be the first path.
diff_
commands
Depending on your merge strategy, you might need to use different diff command.
Buildpipe assumes you are using a merge strategy on the default branch, which is assumed to be master
.
The command for the non-default branch (e.g. when you have a PR up) is:
git log --name-only --no-merges --pretty=format: origin..HEAD
The command for the default branch you merge to is currently:
git log -m -1 --name-only --pretty=format: $BUILDKITE_COMMIT
Requirements
Only curl
is required to download the binary.
Installing buildpipe
You can also install buildpipe in the agent bootstrap script:
curl -Lf -o /usr/local/bin/buildpipe https://github.com/jwplayer/buildpipe-buildkite-plugin/releases/download/v${BUILDPIPE_VERSION}/buildpipe-linux \
&& chmod +x /usr/local/bin/buildpipe
Your initial pipeline would need to pass the options as environment variables with prefix BUILDKITE_PLUGIN_BUILDPIPE_
. For example:
steps:
- label: "buildpipe"
command: buildpipe
env:
BUILDKITE_PLUGIN_BUILDPIPE_DYNAMIC_PIPELINE: path/to/dynamic_pipeline.yml
BUILDKITE_PLUGIN_BUILDPIPE_LOG_LEVEL: debug
Troubleshooting
Buildpipe is incorrectly showing project as changed
Buildkite doesn't by default do clean checkouts. To enable clean
checkouts set the BUILDKITE_CLEAN_CHECKOUT
environment variable. An
example is to modify the pre-checkout hook,
.buildkite/hooks/pre-checkout
:
#!/bin/bash
set -euo pipefail
echo '--- :house_with_garden: Setting up pre-checkout'
export BUILDKITE_CLEAN_CHECKOUT="true"
Testing
make test
License
Apache v2