/piro

An innovative CI/CD system for the Bhojpur.NET Platform that could leverage Kubernetes, Docker, and NanoVMs for running applications or services as managed jobs.

Primary LanguageGoMIT LicenseMIT

Bhojpur Piro - Continuous Integration

A pre-integrated CI/CD system driving many web scale applications or services running over the Bhojpur.NET Platform. On one side, it uses any standard Git repository and drives the Jobs using either a Kubernetes cluster of Docker containers or MicroVMs or NanoVMs on the other side. It knows no pipelines, just the Jobs and each Job could be a Unikernel application and Kubernetes Pod. What you do in that Unikernel or Pod is up to you. We do not impose any "declarative pipeline syntax" or some groovy scripting language. Instead, the Bhojpur Piro jobs have managed to run Node, Go, or Bash scripts in the production environments.

Any standard Git repository (e.g. Bhojpur Seam) could be integrated with the Bhojpur Piro. Though, it is present already in your Bhojpur.NET Platform instance, but different variants are built already to suit requirments of specific industry sector.

Installation

The easiest way to install the Bhojpur Piro is using its Helm chart. Clone this Git repository, cd into helm/, and install using following steps

helm dep update
helm upgrade --install piro .

Git-hoster Integration

The Bhojpur Piro integrates with standard Git hosting platforms using its plugin system. Currently, Bhojpur Piro ships with support for GitHub only (plugins/github-repo and plugins/github-trigger).

To add support for other Git hoster, the github-repo plugin is a good starting point.

GitHub

To use the Bhojpur Piro with the GitHub, you'll need a GitHub application. To create the app, please follow the steps here.

When creating the app, please use following values:

Parameter Value Description
User authorization callback URL https://your-piro-installation.com/plugins/github-integration The /plugins/github-integration
path is important, the domain should match your installation's config.baseURL
Webhook URL https://your-piro-installation.com/plugins/github-integration The /plugins/github-integration path is important,
the domain should match your installation's config.baseURL
Permissions Contents: Read-Only
Commit Status: Read & Write
Issues: Read & Write
Pull Requests: Read & Write
Events Meta
Push
Issue Comments

Configuration

The following table lists the (incomplete set of) configurable parameters of the Bhojpur Piro chart and their default values. The Helm chart's values.yaml is the reference for chart's configuration surface.

Parameter Description Default
repositories.github.webhookSecret Webhook Secret of your GitHub application. See GitHub Setup my-webhook-secret
repositories.github.privateKeyPath Path to the private key for your GitHub application. See GitHub setup secrets/github-app.com
repositories.github.appID AppID of your GitHub application. See GitHub setup secrets/github-app.com
repositories.github.installationID InstallationID of your GitHub application. Have a look at the Advanced page of your GitHub app to find thi s ID. secrets/github-app.com
config.baseURL URL of your Bhojpur Piro installation https://piro.bhojpur.net
config.timeouts.preparation Time a job can take to initialize 10m
config.timeouts.total Total time a job can take 60m
image.repository Image repository bhojpur/piro
image.tag Image tag latest
image.pullPolicy Image pull policy Always
replicaCount Number of cert-manager replicas 1
rbac.create If true, create and use RBAC resources true
resources CPU/memory resource requests/limits
nodeSelector Node labels for pod assignment {}
affinity Node affinity for pod assignment {}

Specify each parameter using the --set key=value[,key=value] argument to helm install.

Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example,

helm install --name my-release -f values.yaml .

Tip: You can use the default values.yaml

OAuth

The Bhojpur Piro does not support OAuth by itself. However, using a standard OAuth Proxy that's easy enough to add. It could leverage Bhojpur IAM for full fledged support for enterprise grade product features.

Setting up Jobs

The Bhojpur Piro jobs are files in your Git repository where one file represents one Job. A Bhojpur Piro job file mainly consists of the PodSpec that will be run. The Bhojpur Piro will add an /application mount to your pod where you'll find the checked out repository the job is running on.

For example:

pod:
  containers:
  - name: hello-world
    image: alpine:latest
    workingDir: /application
    imagePullPolicy: IfNotPresent
    command:
    - sh 
    - -c
    - |
      echo Hello World
      ls

This Job would print Hello World and list all files in the root of the repository.

Checkout Bhojpur Piro's own build job for a more complete example.

Tip: You can use the Bhojpur Piro CLI to create a new Job using piro init job

GitHub events

The Bhojpur Piro starts jobs based on GitHub push events, if the repository contains a .piro/config.yaml file, e.g.

defaultJob: ".piro/build-job.yaml"
rules:
- path: ".piro/deploy.yaml"
  matchesAll:
  - or: ["repo.ref ~= refs/tags/"]
  - or: ["trigger !== deleted"]

The example above starts .piro/deploy.yaml for all tags. For everything else it will start .piro/build-job.yaml.

Log Cutting

The Bhojpur Piro extracts structure from the log output its jobs produce. We call this process log cutting, because the Bhojpur Piro understands logs as a bunch of streams/slices, which have to be de-multiplexed.

The default cutter in the Bhojpur Piro expects the following syntax:

Code Command Description
[someID|PHASE] Some description here Enter new phase Enters into a new phase identified by someID and described by Some description here. All output in this phase that does not explicitely name a slice will use someID as slice.
[someID] Arbitrary output Log to a slice Logs Arbitrary output and marks it as part of the someID slice.
[someID|DONE] Finish a slice Marks the someID slice as done. No more output is expected from this slice in this phase.
[someID|FAIL] Reason Fail a slice Marks the someID slice as failed becuase of Reason. No more output is expected from this slice in this phase. Failing a slice does not automatically fail the job.
[type|RESULT] content Publish a result Publishes content as result of type type

<<<<<<< HEAD

Tip: You can produce this kind of log output using the Bhojpur Piro CLI: piro log <<<<<<< HEAD

<<<<<<< HEAD

Command Line Interface

The Bhojpur Piro sports a powerful continuous integration capability, which can be used to create, list, start and listen to jobs (e.g. Unikernel applications or services) applied in complex data processing.

Direct Installation

The Bhojpur Piro CLI is available in binary format on the release page or using this one-liner:

curl -L https://get.bhojpur.net/piro/install.sh | sh

Simple Usage

The Bhojpur Piro is a very simple GitHub triggered, Unikernel and/or Kubernetes powered CI system.

Usage:
  piro [command]

Available Commands:
  help        Help about any command
  init        Initializes configuration for Bhojpur Piro
  job         Interacts with currently running or previously run jobs
  log         Prints log-cuttable content
  run         Starts the execution of a job
  version     Prints the version of this binary

Flags:
  -h, --help          help for Bhojpur Piro
      --host string   the Bhojpur Piro host to talk to (defaults to PIRO_HOST env var) (default "localhost:7777")
      --verbose       en/disable verbose logging

Use "piro [command] --help" for more information about a command.

Annotations

Annotations are used by your Bhojpur Piro job to make runtime decisions. Bhojpur Piro supports passing annotation in three ways:

  1. From Pull Request description

You can add annotations in the following form to your Pull Request description and the Bhojpur Piro will pick them up

/piro someAnnotation
/piro someAnnotation=foobar
- [x] /piro someAnnotation
- [x] /piro someAnnotation=foobar
  1. From Git commit

The Bhojpur Piro supports same format as above to pass annotations via commit message. The Bhojpur Piro will use the top most commit only.

  1. From Bhojpur CLI
piro run github -a someAnnotation=foobar

Build Source Code

You need Go programming language tools. Issue the following commands in a termainal after gloning this Git repository. It will build and embedd the web user interface and package the PostgreSQL database scripts.

./prepare.sh

then, you can build the Bhojpur Piro CLI using following commands

task build-tools