Automated Deploys Failure on Partially Locked Stages

[vinayhegde90]

Hi Team,

We're using Samson (v2488) for our deployments for 6 apps with 4-5 stages for each. Additionally, we're using the Generic web-hook for automated deploys that we trigger via a scheduled script to all the stages every morning.

While the scheduled deploys for the apps in which no stages haven't been locked complete, we've noticed it fails in those apps wherein at-least 1 (or more) stage has been locked by anyone from our team on the previous day.

Could someone here please help us resolve this? Do let me know if anymore info is required.

[grosser]

So the issue is that a webhook to a locked stage fails ?
Can you explain how to reproduce the issue as simple as possible ... ideally with the dev setup of samson.

[vinayhegde90]

So the issue is that a webhook to a locked stage fails ?
Can you explain how to reproduce the issue as simple as possible ... ideally with the dev setup of samson.

Yes @grosser, we believe that webhooks to locked stages are possibly failing. Below is a step by step guide to reproduce it:

1. Our app, stages & servers structure is as follows:

• app1 with stage dev1 on server1.domain.com
• app1 with stage dev2 on server2.domain.com
• app1 with stage dev3 on server3.domain.com
• app1 with stage devX on serverX.domain.com

2. We deploy app1 on all above stages every morning using web-hooks. Now, if dev1 is locked on the previous day then its corresponding deployment failing is plausible.

3. However, we've noticed if dev1 is locked - web-hook deploys on dev2 & dev3 fail as well.

Can you please help us understand why this might be happening & let us know a solution?

[grosser]

is it 1 webhook that is supposed to deploy all 4 stages ? do you see the failed webhook requests in the logs ? ... what do they show ?

…

On Thu, Mar 14, 2019 at 11:01 PM Vinay Hegde ***@***.***> wrote: So the issue is that a webhook to a locked stage fails ? Can you explain how to reproduce the issue as simple as possible ... ideally with the dev setup of samson. Yes @grosser <https://github.com/grosser>, we believe that webhooks to locked stages are possibly failing. Below is a step by step guide to reproduce it: 1. Our app, stages & servers structure is as follows: - *app1* with stage *dev1* on *server1.domain.com <http://server1.domain.com>* - *app1* with stage *dev2* on *server2.domain.com <http://server2.domain.com>* - *app1* with stage *dev3* on *server3.domain.com <http://server3.domain.com>* - *app1* with stage *devX* on *serverX.domain.com <http://serverX.domain.com>* 1. We deploy *app1* on all above stages every morning using web-hooks. Now, if *dev1* is locked on the previous day then its corresponding deployment failing is plausible. 2. However, we've noticed if *dev1* is locked - web-hook deploys on *dev2* & *dev3* fail as well. Can you please help us understand why this might be happening & let us know a solution? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3241 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAsZ1ikKp3cY7vW4qhEnoPP1XRM2fGaks5vWzcpgaJpZM4bzksq> .

[vinayhegde90]

Yes it's 1 webhook that deploys to all 4 stages via Webhooks mappings, can you clarify where do I check these logs?

[grosser]

most likely stdout or logs/production.log I think I know what happens, the webhook crashes when trying to create a deploy for a locked stage, I'll try to reproduce locally and fix it

…

On Thu, Mar 14, 2019 at 11:57 PM Vinay Hegde ***@***.***> wrote: Yes it's 1 webhook that deploys to all 4 stages via *Webhooks mappings*, can you clarify where do I check these logs? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3241 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAsZwMt41GHAF2zC0V8r8Bl90m6I7Y0ks5vW0RKgaJpZM4bzksq> .

[jason-rutherford]

We have experience this issue as well. If you have webhooks for the same branch to multiple Stages such as:

• origin/master -> UAT-1
• origin/master -> UAT-2
• origin/master -> UAT-3

Then when you lock a Stage such as UAT-2, I expect UAT-1 and UAT-3 deploys to be triggered and UAT-2 not triggered. However, what happens is UAT-1 is triggered and UAT-2 and UAT-3 are not triggered.

[vinayhegde90]

@jason-rutherford : can you mention the version of Samson are you using?

@grosser - could you please specify if this is fixed with your latest commits & the version to which we should update Samson to for resolving this?

[grosser]

v2673 fixes it

[vinayhegde90]

Is v2673 the one specified as latest here or has it not been published yet?

[grosser]

yes, that links to https://github.com/zendesk/samson/tree/935bb353ab9c2dc15bd8a1e119195b8226087bbb which is >v2673

I'll see if I can get the tagging in docker fixed, I think I broke that a few days ago :(

[grosser]

tagging should be fixed now too 🎉

[vinayhegde90]

@grosser if I revisit this, it still doesn't show me version number. Can I safely presume latest would be v2673?

[grosser]

tried fixing it a few times already but it refuses to work :( latest includes v2673

…

On Tue, Mar 19, 2019 at 5:27 AM Vinay Hegde ***@***.***> wrote: @grosser <https://github.com/grosser> if I revisit *this <https://hub.docker.com/r/zendesk/samson/tags>*, it still doesn't show me version number. Can I safely presume latest would be v2673? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3241 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAsZ9jckl6huTyd1ZIPAQb7PMAuRUhaks5vYNe_gaJpZM4bzksq> .

[vinayhegde90]

thanks! shall update Samson to v2673 & post updates here, please keep this thread open.