Atlantis auto merges MR although getting apply error

Question

Atlantis auto merges MR although getting apply error

Opened this issue 4 months ago · 4 comments

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

After pushing changes to my MR, I tried to run atlantis apply -d <state> while the Autoplan is running, and I got these errors from Atlantis -

Plan Error

deleting dir '/atlantis-data/repos/terraform/general/8279/default' before cloning: unlinkat /atlantis-data/repos/terraform/general/8279/default/.git: directory not empty

In the following comment -

Apply Error

`dir "states/dorian-atlantis-test" does not exist`

And eventually -

Automatically merging because all plans have been successfully applied.

If the Auto plan was still running when I ran the apply, I would expect Atlantis to fail the apply command on the reason there is another Atlantis command running. Moreover, even if the apply command took place and got error, I would expect Atlantis not to merge the MR.

I saw some other issues regarding this behavior (#2785 for reference), however I opened another one because I think it's somehow different.

Something important to say - Our Terraform state repository is big with around ~500 states so even shallow cloning takes time to run.

Reproduction Steps

Upload new MR with a state change. Wait for the plan to complete
Upload a simple change to the state so the auto-plan will occur again. Right after commiting the changes and pushing to the branch, run in the MR - atlantis apply -d <your_state>

Logs

atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.214Z","caller":"vcs/gitlab_client.go:120","msg":"Getting modified files for GitLab merge request 8281","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"vcs/gitlab_client.go:141","msg":"GET projects/terraform%2Fgeneral/merge_requests/8281/changes returned: 200","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"events/project_command_builder.go:338","msg":"1 files were modified in this pull request. Modified files: [states/dorian-atlantis-test/bucket.tf]","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"events/project_command_builder.go:396","msg":"got workspace lock","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"events/working_dir.go:106","msg":"clone directory '/atlantis-data/repos/terraform/general/8281/default' already exists, checking if it's at the right commit","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"warn","ts":"2024-09-09T11:51:45.390Z","caller":"events/working_dir.go:121","msg":"will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD: exit status 128: fatal: not a git repository (or any parent up to mount point /)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).\n","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:121\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).buildAllCommandsByCfg\n\tgithub.com/runatlantis/atlantis/server/events/project_command_builder.go:399\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/project_command_builder.go:256\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands.func1\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:29\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).buildAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:71\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:26\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:86\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:305\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:223"}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.10.138.157:53058","json":{}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.10.138.157:53056","json":{}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
atlantis-production-2 atlantis {"level":"info","ts":"2024-09-09T11:52:44.020Z","caller":"events/working_dir.go:235","msg":"creating dir '/atlantis-data/repos/terraform/general/8281/default'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"error","ts":"2024-09-09T11:52:44.820Z","caller":"events/instrumented_project_command_builder.go:75","msg":"Error building auto plan commands: running git clone --depth=1 --branch test-automerge-bug-2 --single-branch https://tbot:<redacted>@gitlab.io/terraform/general.git /atlantis-data/repos/terraform/general/8281/default: Cloning into '/atlantis-data/repos/terraform/general/8281/default'...\nwarning: Could not find remote branch test-automerge-bug-2 to clone.\nfatal: Remote branch test-automerge-bug-2 not found in upstream origin\n: exit status 128","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).buildAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:75\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:26\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:86\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:305\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:223"}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.820Z","caller":"vcs/gitlab_client.go:398","msg":"Updating GitLab commit status for 'atlantis/plan' to 'failed'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.820Z","caller":"vcs/gitlab_client.go:504","msg":"Getting GitLab merge request 8281","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.983Z","caller":"vcs/gitlab_client.go:507","msg":"GET /projects/terraform/general/merge_requests/8281 returned: 200","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.983Z","caller":"vcs/gitlab_client.go:426","msg":"Head pipeline found for merge request 8281, source 'push'. refTarget 'test-automerge-bug-2'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.074Z","caller":"vcs/gitlab_client.go:468","msg":"POST /projects/terraform/general/statuses/e02ad5ce4fd5d6eee5d8cc0ce6eb306d5ef12ca6 returned: 201","json":{"repo":"terraform/general","pull":"8281","attempt":1,"max_attempts":10,"repo":"terraform/general","commit":"e02ad5ce4fd5d6eee5d8cc0ce6eb306d5ef12ca6","state":"failed"}}
atlantis-production-2 atlantis {"level":"error","ts":"2024-09-09T11:52:45.074Z","caller":"events/pull_updater.go:18","msg":"running git clone --depth=1 --branch test-automerge-bug-2 --single-branch https://tbot:<redacted>@gitlab.io/terraform/general.git /atlantis-data/repos/terraform/general/8281/default: Cloning into '/atlantis-data/repos/terraform/general/8281/default'...\nwarning: Could not find remote branch test-automerge-bug-2 to clone.\nfatal: Remote branch test-automerge-bug-2 not found in upstream origin\n: exit status 128","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*PullUpdater).updatePull\n\tgithub.com/runatlantis/atlantis/server/events/pull_updater.go:18\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:91\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:305\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:223"}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.074Z","caller":"vcs/gitlab_client.go:175","msg":"Creating comment on GitLab merge request 8281","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.203Z","caller":"vcs/gitlab_client.go:184","msg":"POST /projects/terraform/general/merge_requests/8281/notes returned: 201","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.203Z","caller":"events/post_workflow_hooks_command_runner.go:53","msg":"post-hooks configured, running...","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.203Z","caller":"events/post_workflow_hooks_command_runner.go:59","msg":"got workspace lock","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.206Z","caller":"events/working_dir.go:106","msg":"clone directory '/atlantis-data/repos/terraform/general/8281/default' already exists, checking if it's at the right commit","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"warn","ts":"2024-09-09T11:52:45.209Z","caller":"events/working_dir.go:121","msg":"will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD: exit status 128: fatal: not a git repository (or any parent up to mount point /)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).\n","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:121\ngithub.com/runatlantis/atlantis/server/events.(*DefaultPostWorkflowHooksCommandRunner).RunPostHooks\n\tgithub.com/runatlantis/atlantis/server/events/post_workflow_hooks_command_runner.go:62\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:225"}
atlantis-production-2 atlantis {"level":"info","ts":"2024-09-09T11:52:45.212Z","caller":"events/working_dir.go:235","msg":"creating dir '/atlantis-data/repos/terraform/general/8281/default'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"error","ts":"2024-09-09T11:52:46.010Z","caller":"events/command_runner.go:228","msg":"Error running post-workflow hooks running git clone --depth=1 --branch test-automerge-bug-2 --single-branch https://tbot:<redacted>@gitlab.io/terraform/general.git /atlantis-data/repos/terraform/general/8281/default: Cloning into '/atlantis-data/repos/terraform/general/8281/default'...\nwarning: Could not find remote branch test-automerge-bug-2 to clone.\nfatal: Remote branch test-automerge-bug-2 not found in upstream origin\n: exit status 128.","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:228"}

Environment details

Atlantis version: atlantis v0.28.5 (commit: 92d10ec) (build date: 2024-07-17T17:07:30.148Z)
Deployment method: helm

Answer 1 · 2024-11-11T10:04:22.000Z

@dorian-tsorfy can I ask - do you have policy checks enabled?

Answer 2 · 2024-11-11T11:07:24.000Z

@dorian-tsorfy can I ask - do you have policy checks enabled?

Yes I have

Answer 3 · 2024-11-14T09:50:29.000Z

I believe that is your issue - we experience the same problems only after enabling policies. Also see: #4861

Answer 4 · 2024-11-14T12:39:56.000Z

Hey @Nordle
I looked at the reference issue you provided but it's quite not the same issue. We're using policy checks from day1 and everything works great from that angle.

The issue I described here occurs when I interfere Atlantis in a way that I manually try to run atlantis apply right after I pushed changes to my MR and Atlantis should Auto-plan on the state.
Maybe it's because the repo is huge and it takes time to Atlantis clone the repo and at that exact time I ran Terraform commands.