Is it possible to require deploys in a specific order?
johnseekins-pathccm opened this issue · 30 comments
Details
Let's say we have two environments: environment_targets: Staging,Production. This works fine and deploys work against both environments just fine. But what if we want to ensure that a user deploys to Staging before they're allowed to deploy to Production? Essentially, is there a way to, similar to requiring reviews, require an environment be successful before another environment can be deployed to?
👋 Hey @johnseekins-pathccm, thanks for opening this issue! This is not the first time I have heard this request and I think its something that warrants a deep dive now. I will schedule some time to investigate if this can be done, and if so... implement it. Keep some 👀 on this issue and I'll update it as I go! It might not be right away as my schedule is pretty tight but I'll get to it when I'm able. Thank you! 🙇
@johnseekins-pathccm I was actually able to complete this faster than expected. I have published a pre-release v9.8.0. Would you be able to test this out and let me know how it works? Thanks! 🙇
Sorry for the bad message. This works just fine. Thanks for the quick turn-around!
Actually...no. The order enforcing works too well:
For some additional clarification:
sha matches in both the successful Staging deploy and the failed Production deploy.
@johnseekins-pathccm would you be able to copy/paste your actions config for the branch-deploy step here or re-run your job with debug logs and paste those too? Feel free to redact anything that might be considered sensitive and I'll take a look to see what's up.
One thing that I am noticing is that I don't see one of the little green rockets (🚀) in your screenshot. Usually if a branch successfully deploys it will say so on your pull request just below that "Deployment Results ✅" comment. I don't see that in your case so it would make sense that the deployment to Production would fail because Staging was never fully deployed.
Here is an example of it working for me:
Fair point, but here's that rocket:
Hmm that is quite strange then.. Do you have logs or your action config that you can copy/paste here? 🙇
Here's the action config:
- uses: github/branch-deploy@v9.8.0
id: branch-deploy
with:
environment_targets: Staging,Production
enforced_deployment_order: Staging,Production
# default deployment environment
environment: Staging
production_environments: Production
skip_reviews: Staging
skip_completing: true
sticky_locks: true
Ah! I see. So it looks like you are completing the deployments via a custom step with skip_completing: true then?
Are you able to share more details about how you are setting your deployment to active? I bet there is a bug somewhere in there and when branch-deploy goes to check its missing some metadata that's causing the new checks to fail...
Here's more of the code around "finishing" a deploy.
- if: "inputs.noop != 'true'"
name: Set Deployment Status
id: set-status
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
gh api --method POST \
"repos/${{ inputs.repository }}/deployments/${{ inputs.deployment-id }}/statuses" \
-f environment=${{ inputs.environment }} \
-f state=${{ inputs.deployment-status }}
# Remove the trigger reaction added to the user's comment.
- name: Remove Trigger Reaction
id: remove-reaction
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
gh api --method DELETE \
"repos/${{ inputs.repository }}/issues/comments/${{ inputs.comment-id }}/reactions/${{ inputs.initial_reaction_id }}"
# Add a new reaction based on if the deployment succeeded or failed.
- name: Add Reaction
id: add-reaction
uses: GrantBirki/comment@v2
env:
REACTION: ${{ inputs.deployment-status == 'success' && 'rocket' || '-1' }}
GH_TOKEN: ${{ github.token }}
with:
comment-id: ${{ inputs.comment-id }}
reactions: ${{ inputs.deployment-status == 'success' && 'rocket' || '-1' }}
# Add a success comment, including the plan/apply output (if present).
- if: "inputs.deployment-status == 'success'"
name: Add Success Comment
id: success-comment
uses: actions/github-script@v7
env:
GH_TOKEN: ${{ github.token }}
with:
script: |
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `### Deployment Results :white_check_mark:
**${{ inputs.actor }}** successfully ${ ${{ inputs.noop }} === 'true' ? '**noop** deployed' : 'deployed' } branch \`${{ inputs.ref }}\` to **${{ inputs.environment }}**`
})
# Add a failure comment, including the plan/apply output (if present).
- if: "inputs.deployment-status == 'failure'"
name: Add Failure Comment
id: failure-comment
uses: actions/github-script@v7
env:
GH_TOKEN: ${{ github.token }}
with:
script: |
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `### Deployment Results :x:
**${{ inputs.actor }}** had a failure when ${ inputs.noop === 'true' ? '**noop** deploying' : 'deploying' } branch \`${{ inputs.ref }}\` to **${{ inputs.environment }}**`
})
# If the deployment failed, fail the workflow.
- if: "inputs.deployment-status == 'failure'"
name: Fail Workflow
shell: bash
id: fail-workflow
env:
GH_TOKEN: ${{ github.token }}
run: |
echo "There was a deployment problem...failing the workflow!"
exit 1
Which is essentially following the docs https://github.com/github/branch-deploy/blob/main/docs/examples.md#multiple-jobs-with-github-environments
Here's (some of) the output of the staging run:
And the similar prod output:
I think you're right that something isn't getting set somewhere...
It looks like (based again on the docs) that I'm setting state to success or failure, never active.
It looks like (based again on the docs) that I'm setting state to success or failure, never active.
I think you have found the problem! But this leads me to more questions... what is the difference between ACTIVE and SUCCESS for a deployment and which one should we actually be using by this action? I'm going to do some digging to see if I can find out
The GraphQL docs (what the action uses to fetch but not set) indicate that there is indeed an ACTIVE field and that makes the most sense to use: https://docs.github.com/en/graphql/reference/enums#deploymentstate
Well that's fixable. Is success used at all? Or should I do res == "success" ? 'ACTIVE' : 'anythingelse'?
Alrighty, I did some investigating and I think there may be a problem in your workflow somewhere because I cannot replicate this issue in my own dev workflows.
Here are some details that I have found...
The success state is indeed the correct state that should be sent to the GitHub REST API when completing a successful deployment. For failed deployments, the failure state is also the correct one to send. So all is fine here and that isn't the problem.
The ACTIVE value is returned from GitHub's GraphQL API endpoints and is used to determine which deployment for a given environment is the "active" one. All other deployments are marked as "inactive" by comparison. So this value is also fine (used internally by the branch-deploy action here).
I have also done some debugging with helpful comments on this pull request: GrantBirki/actions-sandbox#115. You may find this debugging information useful and also helpful to reference the branch-deploy workflow that I am using to compare it to your to look for bugs.
TL;DR: I actually think things are working as expected and I'm not able to recreate the bug that you are running into.
I'm getting a very different result in API returns:
$ gh api repos/pathccm/noop_tool/deployments/1820147308/statuses
[
{
"url": "https://api.github.com/repos/<org>/deployments/1820147308/statuses/4617553256",
"id": 4617553256,
"node_id": "DES_kwDOHjNddM8AAAABEzpFaA",
"state": "success",
"creator": {
"login": "github-actions[bot]",
"id": 41898282,
"node_id": "MDM6Qm90NDE4OTgyODI=",
"avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github-actions%5Bbot%5D",
"html_url": "https://github.com/apps/github-actions",
"followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
"following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
"gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
"organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
"repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
"events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
"received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
"type": "Bot",
"site_admin": false
},
"description": "",
"environment": "Staging",
"target_url": "",
"created_at": "2024-09-23T14:31:33Z",
"updated_at": "2024-09-23T14:31:33Z",
"deployment_url": "https://api.github.com/repos/<org>/deployments/1820147308",
"repository_url": "https://api.github.com/repos/<org>",
"environment_url": "",
"log_url": "",
"performed_via_github_app": null
},
{
"url": "https://api.github.com/repos/<org>/deployments/1820147308/statuses/4617473865",
"id": 4617473865,
"node_id": "DES_kwDOHjNddM8AAAABEzkPSQ",
"state": "in_progress",
"creator": {
"login": "github-actions[bot]",
"id": 41898282,
"node_id": "MDM6Qm90NDE4OTgyODI=",
"avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github-actions%5Bbot%5D",
"html_url": "https://github.com/apps/github-actions",
"followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
"following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
"gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
"organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
"repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
"events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
"received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
"type": "Bot",
"site_admin": false
},
"description": "",
"environment": "Staging",
"target_url": "https://github.com/<org>/actions/runs/10996318031",
"created_at": "2024-09-23T14:23:22Z",
"updated_at": "2024-09-23T14:23:22Z",
"deployment_url": "https://api.github.com/repos/<org>/deployments/1820147308",
"repository_url": "https://api.github.com/repos/<org>",
"environment_url": "",
"log_url": "https://github.com/<org>/actions/runs/10996318031",
"performed_via_github_app": null
}
]
$ gh api repos/<org>/deployments/1820147308
{
"url": "https://api.github.com/repos/<org>/deployments/1820147308",
"id": 1820147308,
"node_id": "DE_kwDOHjNddM5sfT5s",
"task": "deploy",
"original_environment": "Staging",
"environment": "Staging",
"description": null,
"created_at": "2024-09-23T14:23:21Z",
"updated_at": "2024-09-23T14:31:33Z",
"statuses_url": "https://api.github.com/repos/<org>/deployments/1820147308/statuses",
"repository_url": "https://api.github.com/repos/<org>",
"creator": {
"login": "github-actions[bot]",
"id": 41898282,
"node_id": "MDM6Qm90NDE4OTgyODI=",
"avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github-actions%5Bbot%5D",
"html_url": "https://github.com/apps/github-actions",
"followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
"following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
"gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
"organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
"repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
"events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
"received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
"type": "Bot",
"site_admin": false
},
"sha": "2263c48cdb096b195b5b2bf82ae27d47ca518422",
"ref": "test-ordered",
"payload": {
"type": "branch-deploy",
"sha": "2263c48cdb096b195b5b2bf82ae27d47ca518422"
},
"transient_environment": false,
"production_environment": false,
"performed_via_github_app": null
}
Kinda feels like I need to say ACTIVE instead of success.
ACTIVE is not supported. success is clearly the correct input to the API call. This is baffling. @GrantBirki Do you have the actual graph query you ran so I can validate things on my side?
Edit: Never mind. Found the query in the code.
GraphQL Query:
query ($repo_owner: String!, $repo_name: String!, $environment: String!) {
repository(owner: $repo_owner, name: $repo_name) {
deployments(environments: [$environment], first: 1, orderBy: { field: CREATED_AT, direction: DESC }) {
nodes {
createdAt
environment
updatedAt
id
payload
state
ref {
name
}
creator {
login
}
commit {
oid
}
}
}
}
}Variables:
{
"repo_owner": "<owner>",
"repo_name": "<repo>",
"environment": "production"
}Example results:
{
"data": {
"repository": {
"deployments": {
"nodes": [
{
"createdAt": "2024-09-23T18:08:28Z",
"environment": "production",
"updatedAt": "2024-09-23T18:08:43Z",
"id": "DE_kwDOID9x8M5shzsE",
"payload": "\"{\\\"type\\\":\\\"branch-deploy\\\",\\\"sha\\\":\\\"2a000a896fb9a6b2c1bbd69608d60865be22c515\\\"}\"",
"state": "ACTIVE",
"ref": {
"name": "testing"
},
"creator": {
"login": "github-actions"
},
"commit": {
"oid": "2a000a896fb9a6b2c1bbd69608d60865be22c515"
}
}
]
}
}
}
}When you mentioned "getting a very different result in API returns" I was actually getting the same results in my experiment and it all looks as expected so that should be good 👍
I'm getting a different result:
$ curl -Ss -d @graphql.query -H "Authorization: bearer <>" https://api.github.com/graphql | jq
{
"data": {
"repository": {
"deployments": {
"nodes": [
{
"createdAt": "2024-09-23T19:27:00Z",
"environment": "Staging",
"updatedAt": "2024-09-23T19:39:38Z",
"id": "DE_kwDOHjNddM5sikiB",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
}
},
{
"createdAt": "2024-09-23T19:26:39Z",
"environment": "Staging",
"updatedAt": "2024-09-23T19:31:45Z",
"id": "DE_kwDOHjNddM5sikU8",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
}
},
{
"createdAt": "2024-09-23T19:24:39Z",
"environment": "Staging",
"updatedAt": "2024-09-23T19:26:40Z",
"id": "DE_kwDOHjNddM5sijND",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
}
},
{
"createdAt": "2024-09-23T19:24:39Z",
"environment": "Staging",
"updatedAt": "2024-09-23T19:27:00Z",
"id": "DE_kwDOHjNddM5sijMu",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
}
},
{
"createdAt": "2024-09-23T19:24:33Z",
"environment": "Staging",
"updatedAt": "2024-09-23T19:39:37Z",
"id": "DE_kwDOHjNddM5sijIf",
"payload": "\"{\\\"type\\\":\\\"branch-deploy\\\",\\\"sha\\\":\\\"c3ef9875fcdb78177963039c2943e44a6431a328\\\"}\"",
"state": "ACTIVE",
"ref": {
"name": "test-deploy-again"
},
"creator": {
"login": "github-actions"
},
"commit": {
"oid": "c3ef9875fcdb78177963039c2943e44a6431a328"
}
},
{
"createdAt": "2024-09-23T18:52:39Z",
"environment": "Staging",
"updatedAt": "2024-09-23T18:57:17Z",
"id": "DE_kwDOHjNddM5siPaJ",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
}
},
{
"createdAt": "2024-09-23T18:52:17Z",
"environment": "Staging",
"updatedAt": "2024-09-23T18:57:08Z",
"id": "DE_kwDOHjNddM5siPNI",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
}
},
{
"createdAt": "2024-09-23T18:51:32Z",
"environment": "Staging",
"updatedAt": "2024-09-23T18:52:17Z",
"id": "DE_kwDOHjNddM5siOvK",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
}
},
{
"createdAt": "2024-09-23T18:51:31Z",
"environment": "Staging",
"updatedAt": "2024-09-23T18:52:39Z",
"id": "DE_kwDOHjNddM5siOuf",
"payload": null,
"state": "INACTIVE",
"ref": {
"name": "main"
},
"creator": {
"login": "johnseekins-pathccm"
},
"commit": {
"oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
}
},
{
"createdAt": "2024-09-23T18:51:23Z",
"environment": "Staging",
"updatedAt": "2024-09-23T19:26:11Z",
"id": "DE_kwDOHjNddM5siOqK",
"payload": "\"{\\\"type\\\":\\\"branch-deploy\\\",\\\"sha\\\":\\\"2263c48cdb096b195b5b2bf82ae27d47ca518422\\\"}\"",
"state": "INACTIVE",
"ref": null,
"creator": {
"login": "github-actions"
},
"commit": {
"oid": "2263c48cdb096b195b5b2bf82ae27d47ca518422"
}
}
]
}
}
}
}
You'll notice only some of the values (and never the most recent one) come back as ACTIVE.
I had to slightly change the query to get these results:
query {
repository(owner: \"org\", name: \"repo\") {
deployments(environments: [\"Staging\", \"Production\"], first: 10, orderBy: { field: CREATED_AT, direction: DESC }) {
nodes {
createdAt
environment
updatedAt
id
payload
state
ref {
name
}
creator {
login
}
commit {
oid
}
}
}
}
}
You wouldn't happen to have an environment tag floating around in your workflow would ya?...
If you did, there could be the possibility that it is marking your deployment as inactive after the workflow run finishes:
name: Deployment
on:
push:
branches:
- main
jobs:
deployment:
runs-on: ubuntu-latest
environment: production # <--- this value right here, it doesn't play well with branch-deploy
steps:
- name: deploy
# ...Perhaps the better solution would be to fetch more results and then paginate through them. As it looks like your ACTIVE deployment isn't the very first one returned so it will never be found with the current logic. I'm not entirely sure what flows would place it lower down in the list but I think that is something we should account for and guard against
It's absolutely the case that we were stomping on environment. We wanted those fancy little progress bars in our deployment steps...
I'm testing the deploy with environment: corrected.
This was the problem. All along. Sorry for all the churn. These changes work great!
Now to figure out deploy trains with branch deploys...
We wanted those fancy little progress bars in our deployment steps...
I totally hear ya... I wish that there was more control when using the environment: input as well. If you checkout this comment, you'll find a linked issue with a large discussion around the topic as well.