More Than Certified GitOps MiniCamp 2024

The main purpose of this mini camp is to build a GitOps pipeline to deploy resources, managed by terraform, to AWS using GitHub Actions.

Table of contents

More Than Certified GitOps MiniCamp 2024

Requirements

Expand to see requirements

Section	Task	Self-Reported Status	Notes
Setup
	Main branch is protected	✅
	Cannot merge to main with failed checks	✅
	State is stored remotely	✅
	State Locking mechanism is enabled	✅
Design and Code
	Confirm Account Number	✅	data source post condition
	Confirm Region	✅	variable validation
	Add Default Tags	✅	added to provider block
	Avoid Hardcoded Values	✅
	No plaintext credentials	✅	Environment variables set by OIDC
	Pipeline in GitHub Actions only	✅
Validate
	terraform fmt pre-commit hook	✅	Git Hooks managed by trunk-io
	pre-commit hooks are in repo	✅	Git Hooks managed by trunk-io
Test and Review
	Pipeline works on every PR	✅	`on: pull_request trigger`
	Linter	✅	TFLint configured with aws plugin and deep check
	terraform fmt	✅	See PR 3ware#5
	terraform validate	✅	See PR 3ware#5
	terraform plan	✅	See PR 3ware#5
	Infracost with comment	✅	See PR 3ware#4
	Open Policy Agent fail if cost > $10	✅	See PR 3ware#6
Deploy
	terraform apply with human intervention	✅	Applied when PR is approved
	Deploy to production environment		Currently deploying to development environment
Operate and Monitor
	Scheduled drift detection	✅
	Scheduled port accessibility check
Readme
	Organized Structure	✅
	Explains all workflows	✅
	Link to docs for each action	✅
	Contribution Instructions
	Explains merging strategy	✅
Bonus
	Deploy to multiple environments
	Ignore non-terraform changes	✅	Workflow trigger use paths filter for tf and tfvars files.
	Comment PR with useful plan information	✅	See PR 3ware#7
	Comment PR with useful Linter information	✅	See PR 3ware#5
	Open an Issue if Drifted	✅	See Issue 3ware#20
	Open an issue if port is inaccessible
	Comment on PR to apply	✅	See PR 3ware#32

Workflow

Create feature branch off main
Commit change locally and push to remote
Create a draft pull request that targets the main branch: gh pr create --draft --base main

Important

Pull Requests must be set to draft to prevent CODEOWNER reviewers being assigned until the pull request is ready. This cannot be set by default. See open discussion. Unfortunately this also cannot be automated because action runners, using GITHUB_TOKEN for authentication, are unable to run gh pr ready --undo as the integration is unavailable. See open discussion

The workflow will run through the tests (fmt, validate, TFLint), then run terraform plan and post the plan to the pull request and workflow job summary.
To approve the plan, add the approved label.
When the Workflows have completed, mark the PR as ready to assign a reviewer from CODEOWNERS. (again cannot be automated on a runner)

When to apply?

The debate rumbles on. In this case, because it's just me, apply before merge is fine.

Directories vs Workspaces for multiple environments

Another debate. The best argument I have heard for directories was in the Q&A session on 19/10/2024:

"anyone should be able to cd into a terraform working directory and simply run terraform plan without have to worry about workspaces and variable files"

The workflow currently runs in the development directory, with a view to having a production directory should time allow.

Branching Strategy

---
config:
  theme: base
---
gitGraph
  commit id: "prev" tag: "v1.0.0"
  branch feature
  switch feature
  commit id: "Terraform Changes"
  commit id: "Bug Fix"
  commit id: "Plan Diff fix"
  switch main
  merge feature
  commit id: "new" tag: "v1.1.0"

Diagram

---
config:
  look: handDrawn
  theme: neo
---
flowchart LR
  subgraph Fail
    direction LR
    F("`**Fail Required Checks**
    PR Cannot be merged`")
  end
  subgraph Pass
    direction LR
    P("`**Met Required Checks**
    Merge PR`") -->docs(Run terraform-docs) -->rel(Generate a release)
  end
  subgraph Test
    direction LR
    setup("`**Setup**
    AWS Credentials
    Install and Initialise tofu
    Install and Initialise TFLint
    with AWS Plugin`") -->
    validate{"`**Validate**
    terraform fmt
    terraform validate
    tflint
    Infracost fail if > $10`"} -->|Fail|F
  end
  subgraph Development [Deploy Development Environment]
    direction LR
    devplan(terraform plan)-->AP{"`**Approve Plan**
    via 'approved' label`"} -->|No|F
    AP -->|Yes|devapply(terraform apply) -->testdev("`**Diff Check**
    terraform plan -detailed-exitcode`") -->E{Exit code} -->|2 - Diff|PRC(PR Comment)
  end
  PR(Draft Pull Request) --> Test
  validate -->|Pass|devplan
  E{Exit code} -->|0 - Succeeded|P
  PRC -->F
  F -->|Make changes and resubmit|Test

Workflows

Infracost

Infracost runs on pull requests when they are opened or synchronized. The workflow generates a cost difference of the resources between the main branch and the proposed changes on the feature branch.

This workflow also flags any policy violations defined in infracost-policy.rego. See an example in this pull_request

Terraform CI

Validate

Setup AWS credentials using config-aws-credentials using OIDC to assume a role and set the authentication parameters as environment variables on the runner. This step is required when TFLint deep checking for the AWS rule plugin is enabled.
~~Setup terraform using setup-terraform~~ Not required. terraform v1.9.7 already installed on runner image.
Run terraform fmt
Run terraform init
Run terraform validate
Install TFLint using setup-tflint
Initialise TFLint to download the AWS plugin rules.
Run tflint
Run trunk code quality action; this runs checkov and trivy security checks.
Update the PR comments if any of the steps fail and exit the workflow on failure.

Plan

When a draft pull request is opened, and the Test Terraform job has succeeded - a terraform plan will be run. The workflow uses TF-via-PR. This action adds a high level plan and detailed drop down style plan to the workflow summary and updates the pull request with a comment.

[!NOTE] > plan will run on pull_request events when the test job is successful.

Apply

After terraform plan has been run, assuming the plan is satisfactory, add the 'approved' label to he pull request to approve the plan. The workflow will run again - this time running terraform apply. ~~with plan_parity set, to ensure the plan has not changed~~

I tried using pull_request_review as the apply trigger, but this trigger does not support the paths filter. This means an apply could be triggered when adding non tf files - not ideal. See (3ware#22).

I also tried using issue_comment but, because this runs on the default branch, a diff was always detected between plan and apply - not ideal. See (3ware#29)

Note

Apply will run when the 'approved' label is added and the test workflow is skipped. The test workflow is skipped because it only runs on pull_request events. This has been tested in PR 3ware#19

Diff Check

Following a successful apply, another plan is run to check for any diffs. If a diff is detected, a pull request comment is added and the workflow exits with a failure. If a diff is not detected, the pull request can be merged.

Terraform Docs

Terraform docs will run when the pull request is merged. This only needs to run once, following the apply, and not on every commit to a pull request. Updating the README on every commit generates a lot unnecessary commits and you have to pull the updated README prior to the next push to avoid conflicts.

I use my own Terraform Docs reusable workflow which adds job summaries and verified commits to the terraform-docs gh-action.

Release

Generate a CHANGELOG and version tag using semantic release

To do list

Grafana Port Check
Pull request labels environment
Job matrix / branched for multiple environments
Replace manual terraform commands with tf-via-pr for fmt and validate now this is supported
Raise plan-parity issue with TF-via-PR maintainer

rdhar/gitops-2024