/actions-template-sync

:octocat: Github action for syncing other repositories (templates) with current repository. Any git provider like GitHub (enterprise), GitLab, Gittea,.. are supported for the source repository

Primary LanguageShellMIT LicenseMIT

actions-template-sync

All Contributors

actions-template-sync

Lint

shellcheck

test

test-hooks

test-ssh

test-ssh-gitlab

push-docker

gh-pages-mk-docs

abstract

Synchronise git repositories in an automated manner. Different git providers like GitHub (enterprise), GitLab,.. are supported as the source provider. This can help you e.g. for migration from another git provider to GitHub or if you want to mirror git repositories.

History

It is possible to create repositories within Github with GitHub templates. This is a nice approach to have some boilerplate within your repository. Over time, the template repository will get some code changes. The problem is that the already created repositories won't know about those changes. This GitHub action will help you to keep track of the template changes. The initial author of this repository faced that issue several times and decided to write a GitHub action to face that issue. Because of the nice community, several feature requests helped to go on with the development of the action. Now several other features are supported.

Features

flowchart LR
    github_source("fa:fa-github <b>GitHub</b> source repository <b>[private|public]</b>")
    gitlab_source("fa:fa-gitlab <b>GitLab</b> source repository <b>[private|public]</b>")
    any_source("fa:fa-git <b>Any</b> git provider <b>[private|public]</b>")
    github_target{{"fa:fa-github <b>GitHub</b> target repository <b>[private|public]</b>"}}
    github_source --> |"<b>ssh | PAT | github app</b>"| github_target
    gitlab_source --> |"<b>ssh</b>"| github_target
    any_source --> |"<b>ssh</b>"| github_target
Loading
  • Sync other public or private repository (e.g. template repositories) with the current repository
  • Ignore files and folders from syncing using a .templatesyncignore file
  • many configuration options
  • different lifecycle hooks are supported. This opens the possibility to inject custom code into the workflow with a yaml definition file.
  • different git provider like GitLab, Gittea,.. as source are supported (with ssh). See .github/workflows/test_ssh_gitlab.yml for an example.
  • It is not necessarily needed that source and target repository have the same base history. Because of that reason, it is possible to merge 2 totally different repositories with the help of the action.

Usage

Usage changes depending on whether the template repository is public or private, regardless of the visibility of the current repository.

Public template repository

Add this configuration to a GitHub action in the current repository:

# File: .github/workflows/template-sync.yml

on:
    # cronjob trigger
  schedule:
  - cron:  "0 0 1 * *"
  # manual trigger
  workflow_dispatch:
jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4
        # https://github.com/actions/checkout#usage
        # uncomment if you use submodules within the source repository
        # with:
        #   submodules: true

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: <owner/repo>
          upstream_branch: <target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # optional, no default

You will receive a pull request within your repository if there are some changes available in the template.

Private template repository

If your current repository was created from a private template, there are several possibilities.

1. Using a GitHub app

You can create and use a GitHub App to handle access to the private template repository. To generate a token for your app you can use a separate action like tibdex/github-app-token. You have to set up the checkout step with the generated token as well.

jobs:
  repo-sync:
    runs-on: ubuntu-latest

    steps:
     - name: Generate token to read from source repo # see: https://github.com/tibdex/github-app-token
        id: generate_token
        uses: tibdex/github-app-token@v1
        with:
          app_id: ${{ secrets.APP_ID }}
          private_key: ${{ secrets.PRIVATE_KEY }}

      - name: Checkout
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ steps.generate_token.outputs.token }}

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          github_token: ${{ steps.generate_token.outputs.token }}
          source_repo_path: <owner/repo>
          upstream_branch: <target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # optional, no default

2. Using SSH

You have various options to use ssh keys with GitHub. An example is deployment keys. For our use case, write permissions are not needed. Within the current repository, where the GitHub action is enabled, add a secret (e.q. SOURCE_REPO_SSH_PRIVATE_KEY) with the content of your private SSH key. Make sure that the read permissions of that secret fulfill your use case. Set the optional source_repo_ssh_private_key input parameter. It is also possible to use a different git provider, e.g. GitLab.

jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: ${{ secrets.SOURCE_REPO_PATH }} # <owner/repo>, should be within secrets
          upstream_branch: ${{ secrets.TARGET_BRANCH }} #<target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # optional, no default
          source_repo_ssh_private_key: ${{ secrets.SOURCE_REPO_SSH_PRIVATE_KEY }} # contains the private ssh key of the private repository

3. Using a PAT

⚠️ when the source repository is private using PATs, also the target repository must be private. Else it won't work.

Personal access token is an alternative to using passwords for authentication to GitHub. You can add a kind of password to your GitHub account. You need to set the scopes.

  • repo -> all
  • read:org

pat-scopes

Furthermore, you need to set the access within the source repository to allow GitHub actions within the target repository. As mentioned before (you can see the note in the image) you need to set the target repository to private. settings -> actions -> general.

pat-source-repo-access

example workflow definition

name: actions-template-sync

on:
  # cronjob trigger At 00:00 on day-of-month 1. https://crontab.guru/every-month
  schedule:
  - cron:  "0 0 1 * *"
  # manual trigger
  workflow_dispatch:

jobs:
  test-implementation-job:

    runs-on: ubuntu-latest

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ secrets.CUSTOM_GITHUB_PAT }}

      - name: Test action step PAT
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          github_token: ${{ secrets.CUSTOM_GITHUB_PAT }}
          source_repo_path: ${{ secrets.SOURCE_REPO_PATH }} # <owner/repo>, should be within secrets

Configuration parameters

Variable Description Required [Default]
github_token Token for the repo. Can be passed in using $\{{ secrets.GITHUB_TOKEN }} true
source_repo_path Repository path of the template true
upstream_branch The target branch false <The_remote_default>
source_repo_ssh_private_key [optional] private ssh key for the source repository. see false
pr_branch_name_prefix [optional] the prefix of branches created by this action false chore/template_sync
pr_title [optional] the title of PRs opened by this action. Must be already created. false upstream merge template repository
pr_labels [optional] comma separated list. pull request labels. Must be already created. false
pr_reviewers [optional] comma separated list of pull request reviewers. false
pr_commit_msg [optional] commit message in the created pull request false chore(template): merge template changes :up:
hostname [optional] the hostname of the repository false github.com
is_dry_run [optional] set to true if you do not want to push the changes and not want to create a PR false
is_allow_hooks [optional] set to true if you want to enable lifecycle hooks. Use this with caution! false false
is_not_source_github [optional] set to true if the source git provider is not GitHub false false
git_user_name [optional] set the committer git user.name false ${GITHUB_ACTOR}
git_user_email [optional] set the committer git user.email false github-action@actions-template-sync.noreply.${SOURCE_REPO_HOSTNAME}
git_remote_pull_params [optional] set remote pull parameters false --allow-unrelated-histories --squash --strategy=recursive -X theirs

Docker

There are docker images available. Please checkout How to use docker for details.

Example

This repo uses this template and this action from the marketplace. See the definition here.

If you look for a more detailed guide you can have a look at

Trigger

You can use all triggers which are supported for GitHub actions

Ignore Files

Create a .templatesyncignore file. Just like writing a .gitignore file, follow the glob pattern in defining the files and folders that should be excluded from syncing with the template repository.

It can also be stored inside .github folder.

Note: It is not possible to sync also the .templatesyncignore itself. Any changes from the template repository will be restored automatically.

Remark reading the gitglossary (pathspec section) you see a slight difference to the .gitignore file when you like to disable files you need to use :!. E.g. when you like to disable the sync for all files with exceptions, you need to do smth like

:!newfile-1.txt
*

Lifecycle hooks

Different lifecycle hooks are supported. You need to enable the functionality with the option is_allow_hooks and set it to true ⚠️ use this functionality with caution. You can use one of the available docker images to test it out. With great power comes great responsibility.

In addition, you need a configuration file with the name templatesync.yml within the root of the target repository.

The following hooks are supported (please check docs/ARCHITECTURE.md for a better understanding of the lifecycles).

  • install is executed after the container has started and after reading and setting up the environment.
  • prepull is executed before the code is pulled from the source repository
  • precommit is executed before the code is commited
  • prepush is executed before the push is executed, right after the commit
  • prepr is executed before the PR is done

Remark The underlying OS is defined by an Alpine container. E.q. for the installation phase you need to use commands like apk add --update --no-cache python3

Schema and example for the templatesync.yml

hooks:
  install:
    commands:
      - apk add --update --no-cache python3
      - python3 --version
  prepull:
    commands:
      - echo 'hi, we are within the prepull phase'
      - echo 'maybe you want to do adjustments on the local code'
  precommit:
    commands:
      - echo 'hi, we are within the precommit phase'
      - echo 'maybe you want to add further changes before the code is committed'
  prepush:
    commands:
      - echo 'hi, we are within the prepush phase'
      - echo 'maybe you want to add further changes and commits'
  prepr:
    commands:
      - echo 'hi, we are within the prepr phase'
      - echo 'maybe you want to change the code a bit and do another push before creating the pr'

Troubleshooting

  • refusing to allow a GitHub App to create or update workflow .github/workflows/******.yml without workflows permission

    This happens because the template repository is trying to overwrite some files inside .github/workflows/.

    Currently GITHUB_TOKEN can't be given workflow permission. You can grant our workflow with workflows permission using a PAT following the steps below:

    1. Create a PAT with these repository permissions granted: contents:write, workflows:write, metadata:read.

    2. Copy the generated token and create a new secret for your target repository.

    3. Configure the checkout action to use the token in secrets like this:

      # File: .github/workflows/template-sync.yml

on:
    # cronjob trigger
  schedule:
  - cron:  "0 0 1 * *"
  # manual trigger
  workflow_dispatch:
jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ secrets.<secret_name> }}

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: <owner/repo>
          upstream_branch: <target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # optional, no default

  • pull request create failed: GraphQL: GitHub Actions is not permitted to create or approve pull requests (createPullRequest)

    Open your project Settings > Actions > General and select the checkbox Allow GitHub Actions to create and approve pull requests under the Workflow permissions section.

Release Updates

  • ⚠️ starting with version v1 (v1.0.0) the upstream_branch variable default is not main anymore. It is now set to the remote default branch.
  • starting with version v0.5.2-draft the templateversionrc file is not needed anymore. You can delete that file from the target repositories.

Debug

You must create a secret named ACTIONS_STEP_DEBUG with the value true to see the debug messages set by this command in the log. For more information, see "Enabling debug logging."

Comparison with other tools

There are other great tools available within GitHub. Here you can find a comparison.

feature actions-template-sync github-sync git-repo-sync action-template-repository-sync
GitHub action ✔️ ✔️ ❌ ✔️
hooks ✔️ ❌ ❌ ❌
available docker image ✔️ ❌ ❌ ✔️
sync between private and public repo ✔️ PAT,ssh,Github app ✔️ PAT,ssh ❌ local repos ✔️ PAT
sync between 2 private repos ✔️ PAT,ssh,Github app ✔️ PAT,ssh ❌ local repos ✔️ PAT
sync between 2 public repos ✔️ ✔️ ❌ local repos ✔️
two way sync ❌ ✔️ ❌ ❌
Sync from a third-party repo to a Github repo ✔️ ✔️ ❌ local repos ❌
dry run ✔️ ❌ ❌ ✔️
ignore files ✔️ ❌ ❌ ✔️
creates a PR ✔️ ✔️ ❌ ✔️
remarks The action is placed within the target repositories The action is placed within the target repositories CLI meant for local use The action will be based within the base repository with a list of dependent repositories

DEV

The development environment targets are located in the Makefile

make help

For some architectural notes please have a look at the docs

Contributors ✨

Thanks goes to these wonderful people (emoji key):

andy Augustin
andy Augustin
📖 💻 👀 🛡️ 🤔 💬 💡 🖋 📝		 Ugo Pattacini
Ugo Pattacini
📖		 Jose Gabrielle Rivera
Jose Gabrielle Rivera
💻		 P.D. Rittenhouse
P.D. Rittenhouse
🤔		 Daniel Boll
Daniel Boll
🐛		 albertschwarzkopf
albertschwarzkopf
🤔		 Akul Pillai
Akul Pillai
🛡️
Stefan Riembauer
Stefan Riembauer
🤔		 Fabrizio Cacicia
Fabrizio Cacicia
🛡️ 🐛		 Justin Tunis
Justin Tunis
🤔 💻 🐛		 Michael Matos
Michael Matos
🐛		 Gavin Williams
Gavin Williams
🤔		 Marc Siebeneicher
Marc Siebeneicher
🤔 💻 🐛 📖		 Luís Henrique A. Schünemann
Luís Henrique A. Schünemann
🤔 📖 💻
George
George
💬 📖 🤔		 Pedro Rivero
Pedro Rivero
🤔		 Eleanor Bronson
Eleanor Bronson
🤔		 Marvin Osswald
Marvin Osswald
📖		 David Calvert
David Calvert
📖 🐛 💻 🚧 📝		 Andy Airey
Andy Airey
🐛		 Surya Asriadie
Surya Asriadie
🐛
jellllly420
jellllly420
🤔 💬 📖		 Shaun Tabone
Shaun Tabone
💻

This project follows the all-contributors specification. Contributions of any kind are welcome!