Pipeline Trigger Operator

Pipeline Trigger Operator

Overview

The Pipeline Trigger Operator is a tool designed to enhance the automation of software build pipelines. It brings reactive capabilities to your build pipeline by allowing it to react to various events within the kubernetes build environment.

The operator runs on Kubernetes and seamlessly integrates into the GitOps operational framework to enable automated build pipeline creation. It accomplishes this by monitoring events from various resources, including Flux v2 ImagePolicy and GitRepository resources, as well as the pullrequest-operator PullRequest resource. When triggered, it creates a Tekton PipelineRun for a specified Pipeline resource.

Short video with the components overview

AI Engineer

Still wondering what the pipeline-trigger-operator does or what a reactive pipeline actually is? Our AI Engineer got you covered! It's mission is simple and it's purpose is clear: guide you to the highest level of build pipeline efficiency. Our 'Reactive CI/CD Pipeline Engineer', is here to offer expert advice and insights on setting up and managing these pipelines, with a focus on integrating the pipeline-trigger-operator and optionally - the pullrequest-operator. Whether you're a seasoned developer or new to this domain, our AI Engineer is ready to help you navigate and excel in reactive CI/CD. So, feel free to reach out to it with your queries related to our project!

Features

Automated creation of Tekton PipelineRuns in response to events from Flux and Pull Request Operator resources.
Works with GitRepository, ImagePolicy, and PullRequest resources.
Dynamically configurable PipelineRun input parameters using JSON path expressions.
Monitoring support with Prometheus, Grafana, and a dedicated dashboard.

Prerequisites

Before using the Pipeline Trigger Operator, ensure you have the following prerequisites in place:

Flux is installed on your Kubernetes cluster.
The PullRequest Operator is also installed on your cluster.

Installation

You can install the Pipeline Trigger Operator using the following commands in the pipeline-trigger-operator-system namespace:

Without RBAC Proxy:

kubectl apply -f https://github.com/jquad-group/pipeline-trigger-operator/releases/latest/download/release.yaml

Or with RBAC Proxy:

kubectl apply -f https://github.com/jquad-group/pipeline-trigger-operator/releases/latest/download/release-with-rbac-proxy.yaml

Installation with Monitoring Support

The Pipeline Trigger Operator utilizes the following components:

Prometheus Operator
Prometheus
Grafana

Install using the following command:

kubectl apply -f https://github.com/jquad-group/pipeline-trigger-operator/releases/latest/download/release-with-monitoring.yaml

You can import the Grafana dashboard from config/dashboard/dashboard.json to monitor the operator's performance.

Specification

apiVersion: pipeline.jquad.rocks/v1alpha1
kind: PipelineTrigger
metadata:
  name: pipelinetrigger-sample-image
  namespace: jq-example-namespace
spec:
  # Source can be ImagePolicy, GitRepository, or PullRequest
  # The operator subscribes for events from these resources
  source: 
    apiVersion: image.toolkit.fluxcd.io/v1beta2
    kind: ImagePolicy
    name: latest-image-notifier
  # The Tekton PipelineRun
  # See the Tekton API: https://tekton.dev/docs/pipelines/pipeline-api/#tekton.dev/v1beta1.PipelineRun or https://tekton.dev/docs/pipelines/pipeline-api/#tekton.dev/v1.PipelineRun
  pipelineRun: 
    apiVersion: tekton.dev/v1
    kind: PipelineRun
    metadata:
      # Can be either name or generateName
      generateName: microservice-pipeline-
      # This must be the same with the PipelineTriggers namespace
      namespace: jq-example-namespace
      labels:
        app: microservice
    spec:
      pipelineRef:
        name: release-pipeline-go
      serviceAccountName: build-robot
      podTemplate:
        securityContext:
          fsGroup: 0
          runAsGroup: 0
          runAsUser: 0        
      workspaces:
      - name: workspace
        volumeClaimTemplate:
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 128Mi
            volumeMode: Filesystem              
    # The specific input parameters for the pipeline used for the creation of the PipelineRun 
      params:
      - name: "repo-url"
        value: "https://github.com/my-project.git"
      - name: "image-name"
        value: "main" # or JSON path expression, e.g. $.imageName

Usage

Example 1: Build a Microservice Image on Git Push

Create an example namespace:
```
 kubectl create ns jq-example-git
```

Deploy the GitRepository resource to clone a branch from a git repository:

kubectl apply -f examples/create-pipeline-on-git-push/gitrepository.yaml

Deploy the tekton-pipelines.yaml to create a dummy Tekton pipeline, which will be started when the GitRepository detects new commits:
```
kubectl apply -f examples/create-pipeline-on-git-push/tekton-pipelines.yaml
```

Deploy the pipelinetrigger.yaml to trigger the pipeline:

kubectl apply -f examples/create-pipeline-on-git-push/pipelinetrigger.yaml

The PipelineRun will be created automatically, when new git commit is detected.

Example 2: Build a Microservice Image on New Base Image Release

Create an example namespace:
```
kubectl create ns jq-example-image
```
Deploy the ImageRepository resource to fetch tags for a repository in a container registry:
```
kubectl apply -f examples/create-pipeline-on-image-update/imagerepository.yaml
```
Deploy the ImagePolicy resource to select the latest tag from the ImageRepository:
```
kubectl apply -f examples/create-pipeline-on-image-update/imagepolicy.yaml
```
Deploy the tekton-pipelines.yaml to create a dummy Tekton pipeline, which will be started when the ImagePolicy detects new image versions:
```
kubectl apply -f examples/create-pipeline-on-image-update/tekton-pipelines.yaml
```

Deploy the pipelinetrigger.yaml to trigger the pipeline:

kubectl apply -f examples/create-pipeline-on-image-update/pipelinetrigger.yaml

The PipelineRun will be created automatically, when new base image is released.

Example 3: Build a Microservice Image on New Pull Request

Create an example namespace:
```
kubectl create ns jq-example-pr
```
Deploy the PullRequest resource to check for new pull requests for a git repository:
```
kubectl apply -f examples/create-pipeline-on-pr/pullrequest.yaml
```
Deploy the tekton-pipelines.yaml to create a dummy Tekton pipeline, which will be started when the GitRepository detects new commits:
```
kubectl apply -f examples/create-pipeline-on-pr/tekton-pipelines.yaml
```

Deploy the pipelinetrigger.yaml to trigger the pipeline:

kubectl apply -f examples/create-pipeline-on-pr/pipelinetrigger.yaml

The PipelineRun will be created automatically, when new pull requests are detected.

Advanced Usage

Dynamic Input Parameters

The operator accepts JSON path expressions for dynamic input parameters, enabling you to extract and use specific data from events.

Example: Starting a PipelineRun on Events from PullRequest

Given the following Github response:

{"id":1431803478,"number":8,"state":"open","locked":false,"title":"simple pr","created_at":"2023-07-12T18:37:32Z"}

You can use the expression $.title in a PipelineTrigger to extract the title like this:

params:
- name: "pullrequest-title"
  value: $.title

The whole Github/Bitbucket response can be shown, by using kubectl describe pullrequest.

Example: Starting a PipelineRun on Events from GitRepository

You can use the following JSON expressions in the arguments of a PipelineRun:

$.branchName - The current branch name

$.branchId - The current commit ID

$.repositoryName - The current repository name

Example: Starting a PipelineRun on Events from ImagePolicy

You can use the following JSON expressions in the arguments of a PipelineRun:

$.imageName - The current image name

$.imageVersion - The current image version

$.repositoryName - The current repository name

Dual Cluster Deployment

If you plan to deploy the operator in a dual-cluster setup and wish to evenly distribute PipelineRuns between the clusters, you can achieve this by configuring the Controller Deployment with the following arguments:

      containers:
      - name: manager
        command:
        - /manager
        args:
        - --leader-elect
        - --second-cluster
        - --second-cluster-address=xxx
        - --second-cluster-bearer-token=xxx

In this dual-cluster configuration, the operator will efficiently manage and distribute PipelineRuns across the specified clusters.

Development and Contribution

Build and run locally

In order to build the project, run make.

Run the tests with make test.

If the API source files change, run make manifests.

Run the project locally by make deploy.

Build the container image

Build the container image using docker build . --tag pipeline-trigger-operator (implicitly builds the operator)

Build and deploy the changed version by running the following command from the config/default directory:

kustomize build . | kubectl apply -f -