Introduction

This project enables syncing GitHub Issues to a GitHub Project. It can be used either as a Service or a GitHub Action.

Before starting to work on this project, we recommend reading the Implementation section.

How it works

The following events trigger the synchronization of an issue into the project targetted by a Rule:

issues.opened
- Happens when a new issue is created in a repository
issues.reopened
- Happens when a new issue is reopened in a repository
issues.labeled
- Happens when a label is added to an issue

Service

The github-issue-sync service implements a GitHub App which is started by the main entrypoint; consult the Dockerfile for running the server or docker-compose.yml for the whole application.

It is composed of

A web server for receiving GitHub Webhook events via HTTP POST
A database for storing Rules

API

An HTTP API is provided for the sake of enabling configuration at runtime. The following sections will showcase examples of how to use said API through curl.

All API calls are protected by tokens which should be registered by the Create token endpoint.

Create a rule

POST /api/v1/rule/repository/:owner/:name

This endpoint is used to create a Rule for a given repository. A Rule specifies how issues for a repository are synced to a target project. Please check the type of IssueToProjectFieldRuleCreationInput in the source types for all the available fields.

Keep track of the returned ID in case you want to update the rule later; regardless, all IDs can be retrieved at any point by using the listing endpoint.

Unfiltered Rule

If a Rule is specified with no filter, any issue associated with the incoming events will be registered to the board.

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X POST "http://github-issue-sync/api/v1/rule/repository/$owner/$name" \
  -d '{
    "project_number": 1,
    "project_field": "Status",
    "project_field_value": "Done"
  }'

Filtered Rule

Optionally it's possible to specify a jq expression (the cookbook might be helpful) in the "filter" field to be tested against the "issue" object in the webhook's payload.

If a filter is defined, the rule will only be triggered if its filter outputs a non-empty string. For example, if you want the rule to be triggered only for issues which have an "epic" label, define the filter as follows:

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X POST "http://github-issue-sync/api/v1/rule/repository/$owner/$name" \
  -d '{
    "filter": ".labels[] | select(.name == \"epic\")",
    "project_number": 2,
    "project_field": "Epics",
    "project_field_value": "Todo"
  }'

In the example above, if the issue does not have an "epic" label, jq would not output anything according to the "filter" and thus the rule would not be matched. You can verify this locally:

jq -r -n --argjson input '{"labels":[{"name": "foo"}]}' '$input | .labels[] | select(.name == "epic")'

Update a rule

PATCH /api/v1/rule/:id

This endpoint is parameterized by a Rule ID. As the name implies, it updates an existing rule using the request's JSON payload. Please check the type of IssueToProjectFieldRuleUpdateInput in the source types for all the available fields.

Example: Update the filter for an existing rule whose ID is 123

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X PATCH "http://github-issue-sync/api/v1/rule/123" \
  -d '{
    "filter": ".labels[] | select(.name == \"milestone\")"
  }'

Fetch a rule

GET /api/v1/rule/:id

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X GET "http://github-issue-sync/api/v1/rule/$id"

List rules for a specific repository

GET /api/v1/rule/repository/:owner/:name

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X GET "http://github-issue-sync/api/v1/rule/$owner/$name"

List all rules

GET /api/v1/rule

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X GET "http://github-issue-sync/api/v1/rule"

Delete a rule

DELETE /api/v1/rule/:id

curl \
  -H "x-auth: $token" \
  -X DELETE "http://github-issue-sync/api/v1/rule/:id"

Delete all rules for a specific repository

DELETE /api/v1/rule/repository/:owner/:name

curl \
  -H "x-auth: $token" \
  -X DELETE "http://github-issue-sync/api/v1/rule/$owner/$name"

Create a token

POST /api/v1/token

This API will respond with the newly-created token which later can be deleted.

Note that $API_CONTROL_TOKEN should be used as a token here since normal tokens are not able to create other tokens.

curl \
  -H "x-auth: $API_CONTROL_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST "http://github-issue-sync/api/v1/token" \
  -d '{ "description": "Owned by John Smith from the CI team" }'

Delete a token

DELETE /api/v1/token

curl \
  -H "x-auth: $token" \
  -H "Content-Type: application/json" \
  -X DELETE "http://github-issue-sync/api/v1/token"

GitHub App

The GitHub App is necessary for the application to receive webhook events and access the GitHub API properly.

Follow the instructions of https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Bots/Development/Create-a-new-GitHub-App for creating a new GitHub App.

After creating the app, you should configure and install it (make sure the environment is properly set up before using it).

Configuration

Configuration is done at https://github.com/settings/apps/${APP}/permissions.

Repository permissions

Issues: Read-only
- Allows subscription to the "Issues" event

Organization permissions

Projects: Read & write
- Allows for items to be created in GitHub Projects.

Events subscriptions

Issues
- Events used to trigger syncing for our primary use-case

Installation

Having created and configured the GitHub App, install it in a repository through https://github.com/settings/apps/${APP}/installations.

Setup

Requirements

Node.js for running the application
yarn for installing packages and starting scripts
- If it's not already be bundled with Node.js, install with npm install -g yarn
jq for the filtering expressions on Rules
postgres for the database See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Setup#postgres

Environment variables

All environment variables are documented in the src/server/.env.example.cjs file. For development you're welcome to copy that file to src/server.env.cjs so that all values will be loaded automatically once the application starts.

Development

Run the application

Set up the GitHub App
Set up the application

During development it's handy to use a smee.io proxy, through the WEBHOOK_PROXY_URL environment variable, for receiving GitHub Webhook Events in your local server instance.
Start the Postgres instance

See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/Setup#postgres (use the variables of .env.cjs for the database configuration)
Run yarn to install the dependencies
Apply all database migrations
Run yarn dev to start a development server or yarn watch for a development server which automatically restarts when you make changes to the source files
Trigger events in the repositories where you've installed the GitHub App (Step 2) and check if it works

Database migrations

Database migrations live in the migrations directory.

Migrations are executed in ascending order by the file name. The format for their files names is ${TIMESTAMP}_${TITLE}.ts.

Apply all pending migrations: yarn migrations:up
Rollback a single migration: yarn migrations:down
Create a new migration: yarn migrations:create [name]

Check the official documentation for more details.

Deployment

Logs

See https://gitlab.parity.io/groups/parity/opstooling/-/wikis/home

Environments

When you push a deployment tag to GitHub, it will be mirrored to GitLab and then its CI pipeline will be run for deploying the app.

The application can be deployed to the following environments:

Production: push a tag with the pattern /^v[0-9]+\.[0-9]+.*$/, e.g. v0.1
Staging: push a tag with the pattern /^stg-v[0-9]+\.[0-9]+.*$/, e.g. stg-v0.1

Manual

The whole application can be spawned with docker-compose up.

For ad-hoc deployments, for instance in a VM, one idea is to use the docker-compose up command in a tmux session. e.g.

tmux new -s github-issue-sync sh -c "docker-compose up 2>&1 | tee -a log.txt"

GitHub Action

A GitHub Action is ran on-demand by creating a workflow configuration in the default branch of the target repository.

Build

Building entails

Compiling the TypeScript code to Node.js modules
Packaging the modules with ncc

Since the build output consists of plain .js files, which can be executed directly by Node.js, it could be ran directly without packaging first; we regardless prefer to use ncc because it bundles all the code, including the dependencies' code, into a single file ahead-of-time, meaning the workflow can promptly start the action without having to install dependencies on every run.

Build steps

Install the dependencies

yarn

Build the artifacts

yarn action:build

Package the action

yarn action:package

See the next sections for trying it out or releasing.

Trial

A GitHub workflow will always clone the HEAD of ${organization}/${repo}@${ref} when the action is executed, as exemplified by the following line:

uses: paritytech/github-issue-sync@branch

Therefore any changes pushed to the branch will automatically be applied the next time the action is ran.

Trial steps

Build the changes and push them to some branch
Change the workflow's step from paritytech/github-issue-sync@branch to your branch:

-uses: paritytech/github-issue-sync@branch
+uses: user/fork@branch

Re-run the action and note the changes were automatically applied

Release

A GitHub workflow will always clone the HEAD of ${organization}/${repo}@${tag} when the action is executed, as exemplified by the following line:

uses: user/github-issue-sync@tag

That behavior makes it viable to release by committing build artifacts directly to a tag and then using the new tag in the repositories where this action is installed.

Release steps

Build the changes and push them to some tag
Use the new tag in your workflows:

-uses: user/github-issue-sync@1
+uses: user/github-issue-sync@2

Workflow configuration

name: GitHub Issue Sync

on:
  issues:
    # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#issues
    types:
      - opened
      - reopened
      - labeled

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - name: github-issue-sync
        uses: paritytech/github-issue-sync@tag
        with:
          # The token needs to have the following permissions
          # - "read:org" is used to read the project's board
          # - "write:org" is used to assign issues to the project's board
          # - "repo" is used to access issues through the API
          token: ${{ secrets.PROJECTS_TOKEN }}

          # The number of the project which the issues will be synced to
          project: 123

          # The name of the project field which the issue will be assigned to
          target-project-field: Team

          # The value which will be set in the field, in this case the team's
          # name
          target-project-field-value: Foo

Install

Having released the code, the final step is to copy the workflow configuration to the .github/workflows folder of projects whose issues need to be synced.

Implementation

The sync is triggered from:

Webhook events in event handlers for the service
- The event listeners are set up in main
CLI entrypoint for the GitHub Action

joao-paulo-parity/github-issue-sync