General Stuff
This repository contains the Concourse setup for Bookbinder-bound books deployed to Cloud Foundry.
The Concourse pipelines live at https://concourse.run.pivotal.io/. If you need to log in, use the 'Pubtools Concourse CI' username and password in Last Pass.
Concourse Pipelines
A Concourse pipeline is a collection of resources and jobs that, together, accomplish some task. More information on concourse pipelines can be found in the concourse docs.
A pipeline for a book is generated based on the config.yml
in the book's
repository. To create a new pipeline:
- Make a new folder with the name of the pipeline
- Make a child folder with the name of the group
- Add a
config.yml
anddeployment-resources.yml
for the group - Generate the
pipeline.yml
and upload it to concourse
Given the following example config.yml:
---
- book: <book-org>/<book-repo-name>
book_branch: <optional book branch (defaults to master)>
deployments:
- name: staging
depends_on: bind
trigger: true
- name: production
depends_on: staging
trigger: false
The resulting pipeline will look like:
----------
[this repo] - - - - - - | | ----------- --------------
| | | | | |
[book] ---------------- | | ------- | staging | - - - - | production |
| bind | | deploy | | deploy |
[book credentials] ---- | | | | | |
| | ----------- --------------
[book section] -------- | |
----------
The incoming lines indicate whether a change will trigger the job. Dashed will not trigger, and solid line will trigger.
The bind job
The bind job is responsible for creating the bound book for deployment by later jobs.
The book is saved to s3 in the "concourse-interim-steps" bucket.
The deploy jobs
A deploy job is responsible for deploying the bound book onto Cloud Foundry.
-
name
- both the name displayed on the Concourse job and maps to the key underenv
in thecredentials.yml
of the credentials repo for the book. -
depends_on
- adds apassed
constraint on inputs for the given job. -
trigger
- whether or not to automatically run the deploy when previous steps complete.
Using this repo
Getting set up
Requirements
Make sure that you have saved the credentials.yml from LastPass. Within the 'Shared-PubTools' folder there is an item called 'pubtools-concourse-credentials.yml'. The contents of this note should be saved into pipelines/credentials.yml
.
rake newb
will download fly
from the Concourse instance and do any other initial setup.
This can also be done independently with rake 'bin/fly'
.
Generate pipeline.yml
- Run
rake fly:login
if necessary using the same Pubtools Concourse CI credentials from LastPass - Run
rake scheme:update_all[<pipeline>]
- Run
rake fly:set_pipeline[<pipeline>]
More fine-grained control over generating your pipeline
- Run
rake scheme:update[<pipeline>/<group>]
- Run
rake pipeline:update[<pipeline>]
Add a Repo to a Pipeline
Prerequisite:
The repo must be in the config.yml
for the book, for example docs-book-pivotalcf
. You must push this change in order for the commands in this procedure to work.
- If the repo is private, add
pubtools-docs-helper
as an admin for the repo on github. All github access is done through this pubtools bot on github now. - Navigate to the
concourse-scripts-docs
repo in your workspace and pull the master branch. - Check to see that your
concourse-scripts-docs
directory contains a subdirectorypipelines
with a filecredentials.yml
. Create the file and directory if they do not exist. - Log in to LastPass and click on 'pubtools-concourse-credentials.yml' in the 'Shared-PubTools' folder. Paste the contents into
pipelines/credentials.yml
. If you have done this step before, you may not need to repeat it. - Run
rake fly:login
and use the Pubtools Concourse CI credentials from Lastpass. - Run
rake scheme:update[PIPELINE/GROUP]
. The following example updates the cf-current pipeline and the pcf group:rake scheme:update[cf-current/pcf]
. If you are unsure about the pipeline or group name, see https://pubtools.ci.cf-app.com. The pipelines are listed under the hamburger icon, for example click ondata-docs
, thengemfire
to obtainrake scheme:update[data-docs/gemfire]
. The PIPELINE isdata-docs
and the GROUP isgemfire
. - Run
rake fly:set_pipeline[PIPELINE]
. - Type 'y', for yes, when prompted to review configuration changes.
- Add, commit, and push the resulting changes to the
concourse-scripts-docs
repo. - Go to checkman to confirm that the next build displays the new repo listed in the pipeline.
Delete a Job from a Pipeline
Follow the Add a Repo sequence, but also delete job-specific directories:
-
Edit
PIPELINE-GROUP/config.yml
, removing entries for the job. -
Edit
PIPELINE/deployment-resources.yml
, removing any job resources that are no longer needed. -
Delete job-specific directories, the ones that contain
*.plan
files. -
Update the pipeline:
$ rake scheme:update[PIPELINE/GROUP] $ rake pipeline:update[PIPELINE] $ rake fly:set_pipeline[PIPELINE]
-
Type 'y', for yes, when prompted to review configuration changes.
-
Add, commit, and push the resulting changes to the
concourse-scripts-docs
repo.
Troubleshooting
Slack Pubtools via the #cf-pubtools
channel.
File Structure
concourse-scripts-docs
|- pipeline-name
|- deployment-resources.yml
|- pipeline.yml
|- pipeline-group-1
| |- config.yml
| |- pipeline-job-1
| | |- plan.yml
| | |- sources.yml
| | |- task.yml
| |- pipeline-job-2
| |- plan.yml
| |- sources.yml
| |- task.yml
|- pipeline-group-2
|- config.yml
|- pipeline-job-1
| |- plan.yml
| |- sources.yml
| |- task.yml
|- pipeline-job-2
|- plan.yml
|- sources.yml
|- task.yml
plan.yml
- Includes the steps for a Concourse job- does not include the name of the job
- the pipeline will mark these jobs as serial, so that it will not run more than one at a time
- generated by
rake scheme:update
orrake scheme:update_all
sources.yml
- Includes the incoming resources- except bundle
- generated by
rake scheme:update
orrake scheme:update_all
task.yml
- Includes the whole task configuration for a Concourse task- generated by
rake scheme:update
orrake scheme:update_all
- generated by
config.yml
- Config file for generating the pipelinedeployment-resources.yml
- Extra resources used for deployingpipeline.yml
- The generated pipeline.yml to upload to Concourse
Docker
To update the Docker images, modify docker/Dockerfile and then run the following commands in that directory:
Prerequisites:
Install Docker for Mac.
Steps:
- Run the Docker for Mac app and confirm that the Docker daemon is running:
$ docker run hello-world
- In a terminal, navigate to the
~/workspace/concourse-scripts-docs/docker
directory. - Build a new Docker image from the modified Dockerfile with the tag
pubtools/bookbinder-8.1
:
$ docker build -t pubtools/bookbinder-8.1 .
- Log in to Docker with the credentials in LastPass under the Shared-PubTools folder, listed as "Pubtools Docker Hub":
$ docker login
- Push your new Docker image to Docker Hub:
$ docker push pubtools/bookbinder-8.1