Get docker image running in github action

Question

Get docker image running in github action

stevie- opened this issue 9 months ago · 10 comments

Describe the bug

We are trying to run pre-commit/action within github action.
To make sure dependencies are installed we use the managed docker image in the github action.
Despite other pre-commit hooks, pre-commit-terraform doesn't take care of tool installation (probably due to multiple tool approach)

But we get errors runing pre-commit inside this image.

Snippet from the github action logs

Run pre-commit run --show-diff-on-failure --color=always --all-files
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /github/home/.cache/pre-commit/pre-commit.log
Error: Process completed with exit code 1.

Why don't you use the docker image within your gha workflow? https://github.com/antonbabenko/pre-commit-terraform/blob/master/.github/workflows/pre-commit.yaml

How can we reproduce it?

our workflow looks like this.
We added the uid:gid to make sure permissions are aligned with github actions, but this makes no difference.

name: pre-commit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  get-gha-parameters:
    runs-on: ubuntu-latest
    outputs:
      uid: ${{ steps.get.outputs.uid }}
      gid: ${{ steps.get.outputs.gid }}
    steps:
    - name: Get UID and GID
      id: get
      run: |
        echo "uid=$(id -u) gid=$(id -g)"
        echo "uid=$(id -u)" >> $GITHUB_OUTPUT
        echo "gid=$(id -g)" >> $GITHUB_OUTPUT
  pre-commit:
    runs-on: ubuntu-latest
    needs: get-gha-parameters
    container:
      image: ghcr.io/antonbabenko/pre-commit-terraform:latest
      env:
        USERID: ${{ needs.get-gha-parameters.outputs.uid }}:${{ needs.get-gha-parameters.outputs.gid }}
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v5
    - uses: pre-commit/action@v3.0.1

Environment information

OS:

Current runner version: '2.315.0'

docker info:

command output


 /usr/bin/docker version --format '{{.Server.APIVersion}}'
  '1.43'
  Docker daemon API version: '1.43'
  /usr/bin/docker version --format '{{.Client.APIVersion}}'
  '1.43'
  Docker client API version: '1.43'




Docker image tag/git commit:
Digest: sha256:802440d81ee1409184dbc91425319814c42495e04f41e95f9f47c18724533284

ghcr.io/antonbabenko/pre-commit-terraform:latest


Tools versions. Don't forget to specify right tag in command -

TAG=latest && docker run --entrypoint cat pre-commit:$TAG /usr/bin/tools_versions_info


check managed image contents

.pre-commit-config.yaml:

file content
repos:
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.77.3
    hooks:
      - id: terragrunt_fmt
      - id: terraform_fmt
      - id: terraform_tflint
        args:
          - --args=--config=__GIT_WORKING_DIR__/.tflint.hcl
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: end-of-file-fixer
      - id: trailing-whitespace
  - repo: https://github.com/commitizen-tools/commitizen
    rev: v3.1.0
    hooks:
      - id: commitizen
        stages:
          - commit-msg

Answer 1 · 2024-04-09T14:28:53.000Z

Are you sure that it's the pre-commit-terraform container what produces the error?
From what I see, the git gets installed into pre-commit-terraform container image: https://github.com/antonbabenko/pre-commit-terraform/blob/master/Dockerfile#L219-L221 (the pre-commit is also installed into the image)
And the Run pre-commit run --show-diff-on-failure --color=always --all-files is from pre-commit itself: https://github.com/pre-commit/action/blob/main/action.yml#L19

Would you please try and run pre-commit-terraform without pre-commit/action step and see whether the error persists?

Probably related: actions/runner#3000

Answer 2 · 2024-04-09T16:19:27.000Z

Why don't you use the docker image within your gha workflow? https://github.com/antonbabenko/pre-commit-terraform/blob/master/.github/workflows/pre-commit.yaml

Because to check bash hooks you need different hooks and deps than we provide in this repo.
Also, pre-commit GHA has existed for much longer than the Docker image was introduced.

Answer 3 · 2024-04-09T16:23:25.000Z

btw, if that will help you, that's what I use as GHA in my infra

name: Common issues check

on: [pull_request]

env:
  # Prevent GH API rate-limit issue
  GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

jobs:
  pre-commit:
    runs-on: [self-hosted]
    container: python:3.12-slim@sha256:36d57d7f9948fefe7b6092cfe8567da368033e71ba281b11bb9eeffce3d45bc6
    steps:
    - name: Install container pre-requirements
      run: |
        apt update
        apt install -y \
            git \
            curl \
            unzip \
            jq \
            shellcheck \
            nodejs # Needed for Terraform installation
        curl -L https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 > /usr/bin/yq &&\
          chmod +x /usr/bin/yq
    - name: Checkout
      uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
      with:
        ref: ${{ github.base_ref }}

    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
    - run: |
        git config --global --add safe.directory /__w/infrastructure/infrastructure
        git fetch --no-tags --prune --depth=1 origin +refs/heads/*:refs/remotes/origin/*

    - name: Get changed files
      id: file_changes
      run: |
        export DIFF=$(git diff --name-only origin/${{ github.base_ref }} ${{ github.sha }})
        echo "Diff between ${{ github.base_ref }} and ${{ github.sha }}"
        echo "files=$( echo "$DIFF" | xargs echo )" >> $GITHUB_OUTPUT

    - name: TFLint cache plugin dir
      uses: actions/cache@ab5e6d0c87105b4c9c2047343972218f562e4319 # v4.0.1
      with:
        path: ~/.tflint.d/plugins
        key: ubuntu-latest-tflint-${{ hashFiles('.tflint.hcl') }}

    - name: Setup TFLint
      uses: terraform-linters/setup-tflint@19a52fbac37dacb22a09518e4ef6ee234f2d4987 # v4.0.0
      with:
        github_token: "${{ secrets.GITHUB_TOKEN }}"

    - name: Init TFLint
      run: tflint --init

    - name: Setup Terraform docs
      env:
        # renovate: datasource=github-releases depName=terraform-docs lookupName=terraform-docs/terraform-docs
        TERRAFORM_DOCS_VERSION: '0.17.0'
      run: |
        curl -L https://github.com/terraform-docs/terraform-docs/releases/download/v${TERRAFORM_DOCS_VERSION}/terraform-docs-v${TERRAFORM_DOCS_VERSION}-linux-amd64.tar.gz > terraform-docs.tgz \
          && tar -xzf terraform-docs.tgz terraform-docs && rm terraform-docs.tgz \
          && chmod +x terraform-docs && mv terraform-docs /usr/bin/

    - name: Setup tfupdate
      env:
        # renovate: datasource=github-releases depName=tfupdate lookupName=minamijoyo/tfupdate
        TFUPDATE_VERSION: '0.8.1'
      run: |
        curl -L https://github.com/minamijoyo/tfupdate/releases/download/v${TFUPDATE_VERSION}/tfupdate_${TFUPDATE_VERSION}_linux_amd64.tar.gz > tfupdate.tgz \
          && tar -xzf tfupdate.tgz tfupdate && rm tfupdate.tgz \
          && chmod +x tfupdate && mv tfupdate /usr/bin/

    - name: Install shfmt
      env:
        # renovate: datasource=github-releases depName=shfmt lookupName=mvdan/sh
        SHFMT_VERSION: '3.8.0'
      run: |
        curl -L https://github.com/mvdan/sh/releases/download/v${SHFMT_VERSION}/shfmt_v${SHFMT_VERSION}_linux_amd64 > shfmt \
        && chmod +x shfmt && mv shfmt /usr/bin/

    - uses: hashicorp/setup-terraform@a1502cd9e758c50496cc9ac5308c4843bcd56d36 # v3.0.0
      with:
        terraform_version: 1.5.4

    - name: Setup OPA
      uses: open-policy-agent/setup-opa@34a30e8a924d1b03ce2cf7abe97250bbb1f332b5 # v2.2.0
      with:
        version: latest

    # Need to success pre-commit fix push
    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
      with:
        fetch-depth: 0
        ref: ${{ github.event.pull_request.head.ref }}
        # Need to trigger pre-commit workflow on autofix commit
        # Guide: https://web.archive.org/web/20210731173012/https://github.community/t/required-check-is-expected-after-automated-push/187545/
        ssh-key: "${{ secrets.GHA_AUTOFIX_COMMIT_KEY }}"

    - name: Execute pre-commit
      uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
      timeout-minutes: 30
      env:
        SKIP: no-commit-to-branch,manual-apply-warning
      with:
        extra_args: --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}

      # Need to trigger pre-commit workflow on autofix commit.
    - name: Push fixes
      if: failure()
      uses: EndBug/add-and-commit@a94899bca583c204427a224a7af87c02f9b325d5 # v9.1.4
      with:
        # Determines the way the action fills missing author name and email. Three options are available:
        # - github_actor -> UserName <UserName@users.noreply.github.com>
        # - user_info -> Your Display Name <your-actual@email.com>
        # - github_actions -> github-actions <email associated with the github logo>
        # Default: github_actor
        default_author: github_actor
        # The message for the commit.
        # Default: 'Commit from GitHub Actions (name of the workflow)'
        message: '[pre-commit] Autofix violations'

Answer 4 · 2024-04-10T06:26:47.000Z

btw, if that will help you, that's what I use as GHA in my infra

I've seen your approach, but these bloats up the gha action for every tool, which is not self-installed via pre-commit. (Which should be solved via the docker image)

Answer 5 · 2024-04-10T06:46:18.000Z

Are you sure that it's the pre-commit-terraform container what produces the error? From what I see, the git gets installed into pre-commit-terraform container image: https://github.com/antonbabenko/pre-commit-terraform/blob/master/Dockerfile#L219-L221 (the pre-commit is also installed into the image) And the Run pre-commit run --show-diff-on-failure --color=always --all-files is from pre-commit itself: https://github.com/pre-commit/action/blob/main/action.yml#L19

Would you please try and run pre-commit-terraform without pre-commit/action step and see whether the error persists?

Probably related: actions/runner#3000

I've changed the workflow to call pre-commit after checkout, but still get this error. Which maybe is related to the error in the github action runner image and git binary.

# snippet
 pre-commit:
    runs-on: ubuntu-latest
    needs: get-gha-parameters
    container:
      image: ghcr.io/antonbabenko/pre-commit-terraform:latest
      env:
        USERID: ${{ needs.get-gha-parameters.outputs.uid }}:${{ needs.get-gha-parameters.outputs.gid }}
    steps:
    - uses: actions/checkout@v4
    - run: |
        pre-commit --version
        pre-commit run -a

Error:

pre-commit 3.7.0
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /github/home/.cache/pre-commit/pre-commit.log
Error: Process completed with exit code 1.

Will try to check if git binary is available.

Answer 6 · 2024-04-10T06:59:26.000Z

Workflow step added with these debug steps:

  pre-commit --version
  which git
  git --version
  echo $PATH
  pre-commit run -a || true
  cat $HOME/.cache/pre-commit/pre-commit.log

result:

pre-commit 3.7.0
/usr/bin/git
git version 2.38.5
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at /github/home/.cache/pre-commit/pre-commit.log
### version information
``
pre-commit version: 3.7.0
git --version: git version 2.38.5
sys.version:
    3.12.0 (main, Dec  1 2023, 01:43:25) [GCC 12.2.1 20220924]
sys.executable: /usr/local/bin/python3
os.name: posix
sys.platform: linux
``
### error information
``
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
``
``
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pre_commit/git.py", line 58, in get_root
    cmd_output('git', 'rev-parse', '--show-cdup')[1].strip(),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pre_commit/util.py", line 117, in cmd_output
    returncode, stdout_b, stderr_b = cmd_output_b(*cmd, **kwargs)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pre_commit/util.py", line 111, in cmd_output_b
    raise CalledProcessError(returncode, cmd, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/usr/bin/git', 'rev-parse', '--show-cdup')
return code: 128
stdout: (none)
stderr:
    fatal: detected dubious ownership in repository at '/__w/dp-cloud-infra/dp-cloud-infra'
    To add an exception for this directory, call:
    
    	git config --global --add safe.directory /__w/dp-cloud-infra/dp-cloud-infra
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pre_commit/error_handler.py", line 73, in error_handler
    yield
  File "/usr/local/lib/python3.12/site-packages/pre_commit/main.py", line 373, in main
    _adjust_args_and_chdir(args)
  File "/usr/local/lib/python3.12/site-packages/pre_commit/main.py", line 183, in _adjust_args_and_chdir
    toplevel = git.get_root()
               ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pre_commit/git.py", line 64, in get_root
    raise FatalError(
pre_commit.errors.FatalError: git failed. Is it installed, and are you in a Git repository directory?
``

Answer 7 · 2024-04-10T07:47:25.000Z

the fix is to run this, before running pre-commit

git config --global --add safe.directory $GITHUB_WORKSPACE

final workflow ( basically rebuilding pre-commit action without input parameter )

name: pre-commit

on:
  pull_request:
  push:
    branches: [main]

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/antonbabenko/pre-commit-terraform:v1.88.4
    steps:
    - uses: actions/checkout@v4
    - name: fix tar dependency
      shell: bash
      run: |
        apk --no-cache add tar
        # check python modules installed versions
        python -m pip freeze --local
    - uses: actions/cache@v4
      with:
        path: ~/.cache/pre-commit
        key: pre-commit-3|${{ hashFiles('.pre-commit-config.yaml') }}
    - shell: bash
      run: |
        pre-commit --version
        git config --global --add safe.directory $GITHUB_WORKSPACE
        pre-commit run --show-diff-on-failure --color=always || cat $HOME/.cache/pre-commit/pre-commit.log

Suggestion:
To add an example for other and increase your test coverage you could also add a workflow like this in this repo?

Answer 8 · 2024-04-10T12:02:40.000Z

Suggestion:
To add an example for other and increase your test coverage you could also add a workflow like this in this repo?

Good point. Would you be up to contribute via PR please?

Answer 9 · 2024-04-12T13:32:33.000Z

Yeah, safe.directory was introduced by git a few years ago, and caused some distractions across all CIs, including GHA.

Also, pre-commit action in 2.0.3 (which is used in this repo to avoid providing secrets as in #655 (comment)) can commit and push fixes to branch.

It's not very useful just to get information that something is wrong, especially if it can be fixed automatically.

Answer 10 · 2024-05-23T03:08:33.000Z

This issue has been resolved in version 1.90.0 🎉