dreitier/conditional-regex-search-and-replace-action

Executes conditional search and replace operations on a set of files. Strings are replaced by matching regular expressions.

conditional-regex-search-and-replace-action

This GitHub action executes conditional search and replace operations on multiple files. Strings in files are replaced, if they match a given regular expressions.

Originally, this action has been created to update strings - like Docker image tag and Git branches - in Helm charts and Kustomize templates inside of Git repositories used for GitOps projects. When using the environment-per-folder strategy, updating the designated file for one or multiple environments can be cumbersome. If you want to use this action indeed for updating GitOps repositories, please look at this article on how to execute it with repository_dispatch.

Usage

uses: dreitier/conditional-regex-search-and-replace-action
env:
with:
  mappings: "docker_image_tag==main.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex"
  docker_image_tag: "${{ github.sha }}"
  docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""

Mandatory arguments

Argument	Description
mappings	DSL to define search-and-replace operations, see below

Optional arguments

Argument	Default	Description
directory	$CWD	Directory, in which to operate. By default, the current working directory is used.
dump	<none>	If 1, it dumps the provided configuration
if_no_match_fail	0	If the action has not modified any file and if_no_match_fail is 1, it will fail with exit code 3
if_well_known_vars_missing_fail	0	If 1, it fails with exit code 2 if none of docker_image_tag, git_branch or git_tag is provided
updated_file_suffix	<none>	If set, any changes will be written to another file the path of the original file and that suffix
register_custom_regexes	<none>	A comma-separated list of custom regular expressions to register
register_custom_variables	<none>	A comma-separated list of custom variables to register
commit	0	If there has been any modified files (see below, outputs.total_modified_files), it will commit the changes to the Git repository
commit_template	<none>	Template to use for the commit message. You can use the Blade template engine to dynamically specify the message.
committer_name	<none>	Name of committer, only relevant if commit is present
committer_email	<none>	E-mail of committer, only relevant if commit is present
skip-regexes-autodetect	0	Skip auto-detection of regexes for environment variables ending with _REGEX
skip-variables-autodetect	0	Skip auto-detection of variables for environment variables ending with _VAR

Well-known variables and regular expressions

Due the original requirement of this action, you can use the following built-in variables:

Argument	Description
__always__ and __true__	Both variables are automatically set to 1
docker_image_tag	Docker image tag created by upstream repository
docker_image_tag_regex	Regular expression to modify occurences of Docker image tags in globbed files
git_tag	Git tag created by upstream repository
git_tag_regex	Regular expression to modify occurences of Git tags in globbed files
git_branch	Git branch modified in upstream repository
git_branch_regex	Regular expression to modify occurences of Git branch in globbed files

There is no need to use docker_image_tag_regex, git_tag_regex and git_branch_regex, those regular expressions are automatically registered. You can register additional regular expressions as you like.

Registering additional regular expressions

Additional regular expressions are automatically detected if the environment variable has the suffix (_REGEX). If your regular expression has not that suffix, you can use the option register_custom_regexes:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  register_custom_regexes: check_for_customer_myregex
env:
  # this one is automatically detected
  MY_CUSTOM_REGEX: "/a: b/"
  # this is detected by using `register_custom_regexes`
  CHECK_FOR_CUSTOMER_MYREGEX: "/customer: (<customer_number>\d+)/"

To register multiple regular expressions without the _REGEX suffix, use a comma (,) for separation:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  register_custom_regexes: check_for_customer_myregex, project_id_myregex
env:
  CHECK_FOR_CUSTOMER_MYREGEX: "/customer: (?<customer_number>\d+)/"
  PROJECT_MYID_REGEX: "/project: (?<project_id>\d+)/"

Registering additional variables

To provide additional variables aside from docker_image_tag, git_tag and git_branch you have to register those variables.

• Variables are automatically detected for environment variables having the suffix _VAR.
• If your variable cannot have the suffix _VAR, you have to use register_custom_variables.

For referencing the customer_number variable of the previous section, you can use either:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
env:
  # this one is automatically detected by using the _VAR suffix.
  CUSTOMER_NUMBER_VAR: "555"

or

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  register_custom_variables: customer_number
env:
  # custom registered variable
  CUSTOMER_NUMBER: "555"

In both cases, you can reference the variable later on with customer_number. The _VAR suffix is removed.

To register multiple variables without the _VAR suffix, use a comma (,) for separation:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  register_custom_variables: customer_number, project_id
env:
  CUSTOMER_NUMBER: "555"
  PROJECT_ID: "5550555

Git commit message templates

When using commit_template, you can either put in some static string or you can use the Blade template engine to dynamically specify the message. Each of the well-known and custom variables is passed as variable to the template. If you have a custom variable customer_number, you can access the value with {{ $customer_number }}.

Use a static string:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  commit_template: 'A new version has been born.'

Reference a well-known variable:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  docker_image_tag: '1.0.0'
  commit_template: 'chore: Update to Docker image tag {{ $docker_image_tag }}'

Reference a custom variable:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  docker_image_tag: '1.0.0'
  commit_template: 'chore: Update to Docker image tag {{ $docker_image_tag } for customer {{ $customer_number))'
env:
    CUSTOMER_NUMBER_VAR: "555" # or set it to an empty string

Expressions:

uses: dreitier/conditional-regex-search-and-replace-action
with:
  # ...
  docker_image_tag: '1.0.0'
  commit_template: 'chore: Update to Docker image tag {{ $docker_image_tag }}@if(!empty($customer_number)) for customer {{ $customer_number }}@endif'
env:
    CUSTOMER_NUMBER_VAR: "" # or a non-empty string

Mappings

Mappings specify, if one or multiple regular expressions match, how other regular expressions are applied. It's basically

uses: dreitier/conditional-regex-search-and-replace-action
with:
  mappings: '<if_first_regex_match> OR <if_second_regex_match> {THEN_UPDATE_FILES} <change_first_file_group>=<apply_regex_1> {AND} <change_second_file_group>=<apply_regex_1>&<apply_regex_2>'

In each regular expression to apply, you have access to all well-known and custom variables through named capturing groups:

uses: dreitier/conditional-regex-search-and-replace-action
  docker_image_tag: '1.0.0'
  docker_image_tag_regex: 'imageTag: \"(?<docker_image_tag).*)\"'
  mappings: 'docker_image_tag==.* {THEN_UPDATE_FILES} **.yaml=docker_image_tag_regex&my_regex'
env:
  # auto-detect both variable and regex
  CUSTOMER_NUMBER_VAR: "555"
  MY_REGEX: '.*customer: \"(?<customer_number>.*)\".*'

Outputs

Output	Default	Description
total_modified_files	0	Number of modified (target) files.
$your_variable	``	Any of your provded well-known or custom variables

You can reference the output variables in your GitHub Action workflow like this:

jobs:
  my_job_with_dedicated_commit_and_push:
    runs-on: ubuntu-latest
    steps:
      - name: Update repository  
        id: search_and_replace_op
        uses: dreitier/conditional-regex-search-and-replace-action
        with:
          # ...

      # instead of using this action's commit hook, we use the more flexible EndBug/add-and-commit if any file has been modified
      - name: Commit and push
        if: ${{ steps.search_and_replace.outputs.total_modified_files >= 1 }}
        uses: EndBug/add-and-commit@v7
        with:
          author_name: build@internalcom
          author_email: build@internal
          message: "There have been {{ steps.search_and_replace.outputs.total_modified_files }} modified files"

To access e.g. the customer_number from before

jobs:
  my_job_with_dedicated_commit_and_push:
    runs-on: ubuntu-latest
    steps:
      - name: Update repository
        id: search_and_replace_op
        uses: dreitier/conditional-regex-search-and-replace-action
        with:
          docker_image_tag: '1.0.0'
          # ...
        env:
            CUSTOMER_NUMBER_VAR: "555"

      - name: Commit and push
        if: ${{ steps.search_and_replace.outputs.total_modified_files >= 1 }}
        uses: EndBug/add-and-commit@v7
        with:
          author_name: build@internal
          author_email: build@internal
          message: "Customer number {{ steps.search_and_replace.outputs.customer_number }} has {{ steps.search_and_replace.outputs.total_modified_files }} modified files for tag {{ steps.search_and_replace.outputs.docker_image_tag }}"

Examples

Updating strings in multiple files

In a folder structure like this

├── asia
│   ├── dev
│   │   ├── values.yaml
│   └── prod
│       ├── values.yaml
├── eu
│   ├── dev
│   │   └── values.yaml
│   └── prod

the files eu/dev/values.yaml and asia/dev/values.yaml have be modified. The original content of both values.yaml files looks like this:

custom_parameter: custom_value
imageTag: v0.1.0
other_parameter: other_custom_value

We want to update the imageTag value in both values.yaml files from v0.1.0 to v0.2.0:

uses: dreitier/conditional-regex-search-and-replace-action
env:
with:
  mappings: "docker_image_tag==v.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex"
  docker_image_tag: "v0.2.0"
  docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""

After running conditional-regex-search-and-replace-action, both files will look like this:

custom_parameter: custom_value
imageTag: v0.2.0
other_parameter: other_custom_value

Transformed into pseudo code

Sometimes it's easier to read pseudo code instead of skimming through YAML files. The above YAML's definition can be read as

if $docker_image_tag =~ /v.*/ then
    $files = glob_files_with_matcher("**dev/values.yaml")
    
    foreach $files as $file
        foreach $line in $file
            if $line =~ /imageTag: \"(?<docker_image_tag>.*)\"/ then
                $line = "imageTag: \"$docker_image_tag\""
                write_line_to_file($file, $line)
            fi
        endforeach
    endforeach
endif

Do not check a regex for a given pattern

If you do not have to check one or multiple known variable for a regex, you can simply use the well-known variables __always__ or __true__:

with:
  mappings: "__always__ {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex"
  docker_image_tag: "v0.2.0"
  docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""

By using __always__ or __true__ the replacement will be applied, regardless of one of the variable's value.

Updating multiple values at the same time

If you need to apply multiple regexes at the same time, you can chain those regexes with:

with:
  mappings: "docker_image_tag==v.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex"
  git_branch: "feature/165-integrate-kerberos-auth"
  git_branch_regex: "branch: \\\"(?<git_branch>.*)\\\""
  docker_image_tag: "v0.2.0"
  docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""

A values.yaml of this:

custom_parameter: custom_value
imageTag: v0.2.0
branch: old-branch

will be transformed into this:

custom_parameter: custom_value
imageTag: v0.2.0
branch: feature/165-integrate-kerberos-auth

Transformed into pseudo code

if $docker_image_tag =~ /v.*/ then
    $files = glob_files_with_matcher("**dev/values.yaml")
    
    foreach $files as $file
        foreach $line in $file
            if $line =~ /imageTag: \"(?<docker_image_tag>.*)\"/ then
                $line = "imageTag: \"$docker_image_tag\""
                write_line_to_file($file, $line)
            fi
            if $line =~ /branch: \"(?<git_branch>.*)\"/ then
                $line = "git_branch: \"$git_branch\""
                write_line_to_file($file, $line)
            fi
        endforeach
    endforeach
endif

Check, if at least one variable matches

Sometimes you want to update files if at least one of multiple conditions is valid. mappings support a simple {OR} operator.

with:
  mappings: "docker_image_tag==v.* {OR} git_branch==feature\/.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex&git_branch_regex"
  git_branch: "feature/165-integrate-kerberos-auth"
  git_branch_regex: "branch: \\\"(?<git_branch>.*)\\\""
  docker_image_tag: "latest"
  docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""

Transformed into pseudo code

if $docker_image_tag =~ /v.*/ or $git_branch =~ /feature\/.*/ then
    $files = glob_files_with_matcher("**dev/values.yaml")
    
    foreach $files as $file
        foreach $line in $file
            if $line =~ /imageTag: \"(?<docker_image_tag>.*)\"/ then
                $line = "imageTag: \"$docker_image_tag\""
                write_line_to_file($file, $line)
            fi
            if $line =~ /branch: \"(?<git_branch>.*)\"/ then
                $line = "git_branch: \"$git_branch\""
                write_line_to_file($file, $line)
            fi
        endforeach
    endforeach
endif

Multiple mappings

This action's DSL for defining mappings does not feature a {AND} condition in the left-hand part of {THEN_UPDATE_FILES}. Instead, you can chain multiple mappings together. Use De Morgan's law for complex conditions ;-) Each of those mappings will be separately evaluated:

with:
  mappings: "docker_image_tag==v.* {THEN_UPDATE_FILES} **dev/values.yaml=docker_image_tag_regex {NEXT_MAPPING} git_branch==feature\/.* {THEN_UPDATE_FILES} **dev/values.yaml=git_branch_regex"
  git_branch: "feature/165-integrate-kerberos-auth"
  git_branch_regex: "branch: \\\"(?<git_branch>.*)\\\""
  docker_image_tag: "v2.0.0"
  docker_image_tag_regex: "imageTag: \\\"(?<docker_image_tag>.*)\\\""

Transformed into pseudo code

if $docker_image_tag =~ /v.*/ then
    $files = glob_files_with_matcher("**dev/values.yaml")
    
    foreach $files as $file
        foreach $line in $file
            if $line =~ /imageTag: \"(?<docker_image_tag>.*)\"/ then
                $line = "imageTag: \"$docker_image_tag\""
                write_line_to_file($file, $line)
            fi
        endforeach
    endforeach
endif

if $git_branch =~ /feature\/.*/ then
    $files = glob_files_with_matcher("**dev/values.yaml")
    
    foreach $files as $file
        foreach $line in $file
            if $line =~ /branch: \"(?<git_branch>.*)\"/ then
                $line = "git_branch: \"$git_branch\""
                write_line_to_file($file, $line)
            fi
        endforeach
    endforeach
endif

FAQ

Why not using something more GitOps-esk like Argo CD Image Updater?

Using Argo CD Image Updater is totally fine but might have some drawbacks:

• By default, Argo CD Image Updater checks the Docker registries every 2 minutes. There might be a good chance of hitting API request limits, e.g. with Docker Hub.
• Setting up Argo CD Image Updater might be difficult, depending upon the environment.
• Argo CD Image Updater does not support complex search-and-replace operations.

When using conditional-regex-search-and-replace-action, you can either configure Webhooks in your GitOps repository to notify Argo CD or let Argo CD pull the latest version.

Why are you not using Bash/$your_favorite_language_here for this action?

Before introducing this action, I've developed a Bash script for updating various GitOps repositories. Different projects had different requirements: The Bash script was no longer maintainable. Using Laravel Zero and Pest for developing testable GitHub Actions looks fine to me.

But I can do everything with awk & sed.

Just keep using awk & sed, it's fine!

Why looks the DSL like it does?

Due to the nature of complexity and how parameters are passed from the GitHub workflow to single actions, it was the first approach I came up with. I also thought about using Blade templates for doing the mapping stuff.

DSL spec

next_mapping            = "{NEXT_MAPPING}"
if_match_then_execute   = "{THEN_UPDATE_FILES}"
glob                    = $valid_glob
regex                   = $valid_quoted_regex
variable_reference      = (lower_case_chars | digits | '_')+
regex_reference         = (lower_case_chars | digits | '_')+

matches                 = "=="
matcher                 = variable_reference (matches regex)?
or_matcher              = "{OR}"
matchers                = matcher (or_matcher matcher)+

regex_references        = regex_reference ('&' regex_reference)+
assign_to               = '='
replacer                = glob assign_to regex_references
replacers               = replacer (and_replacer replacer)+
and_replacer            = "{AND}"

mapping                 = matchers if_match_then_execute replacers
multiple_mappings       = mapping (next_mapping mapping)+

Support

This software is provided as-is. You can open an issue in GitHub's issue tracker at any time. But we can't promise to get it fixed in the near future. If you need professionally support, consulting or a dedicated feature, please get in contact with us through our website.

Contribution

Feel free to provide a pull request.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.