/concourse-pipelines

Collection of Concourse pipelines for AWS infrastructure provisioning

Primary LanguageShell

Concourse Pipelines

This repository contains a collection of Concourse CI pipelines used for AWS infrastructure provisioning. Each pipeline directory contains a pipeline.yml defining the pipeline, and a settings.yml file containing the expected parameters. The scripts/ci directory contain the scrips used by the templates.

Pre-requisites

To configure a pipeline all settings in the settings.yml must be pre-populated, with sensitive data stored in Vault following Concourse's credential lookup rules. The yaml2vault.py can be used to write secrets from a yaml file to Vault.

To pipeline templates require Concourse to be setup and configured with Vault for secrets management. A vagrant box is provided this repository with a sample Concourse setup for local testing.

Pipeline setup

A pipeline can be setup with the following commands

fly --target lite login -c ${CONCOURSE_URL}
fly --target lite set-pipeline --pipeline ${PIPELINE_NAME} --config pipeline.yml -n --load-vars-from settings.yml
fly --target lite unpause-pipeline --pipeline ${PIPELINE_NAME}

Pipeline Templates

pipeline-terraform-commit

Handles AWS infrastructure provisioning workflow using terraform, Vault (AWS backend), terrascan (for static code analysis), and AWS S3 for storage of terraform plans. The pipeline is triggered through commits to a specified branch.

pipeline-terraform-commit

How it works

Jobs:

  • Build: Makes sure that terraform is formatted and runs terraform plan, storing the output in S3.
  • Test: Runs terrascan to check for any security/operational issues in the code. An email is sent if successful.
  • Provision: Needs to be manually triggered to run terraform apply in all environments with changes. This will create a GitHub release.
  • Versioning: Manual jobs can be triggered to bump the semantic version of the releases. The create_version_branch.sh script helps setup the target repository for versioning.

Setup:

The following resources need to be configured in settings.py

  • Vault for AWS STS token access - The vault-aws-setup.sh script contains a sample configuration of Vault's AWS secret backend.
  • SMTP - Email notifications are sent when a commit has successfully completed the testing job
  • S3 bucket - The terraform plans are stored in S3
  • GitHub - The repository containing the terraform templates needs to be provided in addition to a personal access token and access keys.

pipeline-terraform-pr

Pipeline used to review pull requests on terraform repositories. The pipeline will attempt to do a terraform plan on all environments, and run terrascan. It's triggered by any pull requests to the configured repository.

pipeline-terraform-commit

How it works

Jobs:

  • Build: - Build: Makes sure that terraform is formatted and runs terraform plan for each environment. The output of the terraform fmt command and terraform plan is stored as a comment on the pull request.
  • Test: Runs terrascan to check for any security/operational issues in the code. The output is stored as a comment on the pull request.

Setup:

The following resources need to be configured in settings.py

  • Vault for AWS STS token access - The vault-aws-setup.sh script contains a sample configuration of Vault's AWS secret backend.
  • GitHub - The repository containing the terraform templates needs to be provided in addition to a personal access token and access keys.