Github Actions deployment to Azure stuck to Package deployment using ZIP Deploy initiated

Question

Github Actions deployment to Azure stuck to Package deployment using ZIP Deploy initiated

jeffordray92 opened this issue 2 years ago · 24 comments

When I was trying to deploy on Azure Function App, the deployment on Github Actions seems to be freezing on Package deployment using ZIP Deploy initiated. This goes on for hours, without any success response, hence we are left with no choice but to cancel the operation.

But, when I checked on Azure deployment logs, it seems to be deployed successfully.

What could be the reason for this, and how should we fix it?

Answer 1 · 2023-05-10T08:11:00.000Z

Thanks @jeffordray92 for reporting this. There is a trail of people (me included) reporting the same issue here:
https://learn.microsoft.com/en-us/answers/questions/1261639/github-actions-deployment-stuck-to-package-deploym

Answer 2 · 2023-05-24T09:03:07.000Z

This issue is idle because it has been open for 14 days with no activity.

Answer 3 · 2023-06-06T11:07:29.000Z

I can confirm, it is a huge issue, both deploying dotnet core web app and Azure Functions (both Linux). For some reason, Azure functions take ~5 minutes while web app takes up to several hours., logs show no apparent reason. But when deploying web app from Visual Studio, it takes seconds, not even minutes. All deployments seem to be related to ZIP Packaging and using Kudu service.

**Update: 7/6/23 **
Reverting the webdeploy to v1 (rather than v2) seems to shorten the time significantly, although deployment randomly fails 1 out of 4 times. Increasing Service Plan from B1 to S1 did not help. For debugging, it would be very helpful with more information on, what is causing the problem - attempting to abort the deployment will not help, instead an error will return that a deployment is already in progress, (obviously the use of cancelTokens has not been utilized correctly ?).

Answer 4 · 2023-06-07T10:45:15.000Z

I have experienced the same issue. Logs are listed below.

Run azure/functions-action@v1
  
Successfully parsed SCM credential from publish-profile format.
Using SCM credential for authentication, GitHub Action will not perform resource validation.
(node:1725) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Successfully acquired app settings from function app (with SCM credential)!
Will archive ./function into /home/runner/work/_temp/temp_web_package_19174212886521258.zip as function app content
Will use Kudu https://<scmsite>/api/zipdeploy to deploy since publish-profile is detected.
Setting SCM_DO_BUILD_DURING_DEPLOYMENT in Kudu container to false
Update using context.kuduService.updateAppSettingViaKudu
Response with status code 204
App setting SCM_DO_BUILD_DURING_DEPLOYMENT propagated to Kudu container
Setting ENABLE_ORYX_BUILD in Kudu container to false
Update using context.kuduService.updateAppSettingViaKudu
Response with status code 204
App setting ENABLE_ORYX_BUILD propagated to Kudu container
Package deployment using ZIP Deploy initiated.

Answer 5 · 2023-06-08T07:10:58.000Z

It works now, where it fails multiple times about 20 hours ago.

The entire process took less than one minute to complete when it works.

Run azure/functions-action@v1
Successfully parsed SCM credential from publish-profile format.
Using SCM credential for authentication, GitHub Action will not perform resource validation.
(node:1670) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Successfully acquired app settings from function app (with SCM credential)!
Will archive ./function into /home/runner/work/_temp/temp_web_package_9794794788908359.zip as function app content
Will use Kudu https://<scmsite>/api/zipdeploy to deploy since publish-profile is detected.
Setting SCM_DO_BUILD_DURING_DEPLOYMENT in Kudu container to false
Update using context.kuduService.updateAppSettingViaKudu
Response with status code 204
App setting SCM_DO_BUILD_DURING_DEPLOYMENT propagated to Kudu container
Setting ENABLE_ORYX_BUILD in Kudu container to false
Update using context.kuduService.updateAppSettingViaKudu
Response with status code 204
App setting ENABLE_ORYX_BUILD propagated to Kudu container
Package deployment using ZIP Deploy initiated.
Deploy logs can be viewed at https://00000000-0000-0000-0000-000000000000.scm.azurewebsites.net/api/deployments/00000000-0000-0000-0000-000000000000/log
Successfully deployed web package to App Service.
Restoring SCM_DO_BUILD_DURING_DEPLOYMENT in Kudu container to false
Response with status code 204
Restoring ENABLE_ORYX_BUILD in Kudu container to false
Response with status code 204

Answer 6 · 2023-06-26T18:03:37.000Z

This issue is idle because it has been open for 14 days with no activity.

Answer 7 · 2023-07-03T13:05:06.000Z

This is still an issue, it keeps on hanging, the deployment happens, but there is not termination feedback for the pipeline, but what is interesting is that if you terminate the function app via terraform and re-create it, it works fine initially, I thought that Config vars are making this behave in a weird way, but no.

It hangs like this:

##[debug][GET] https://***.scm.azurewebsites.net:443/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2023-07-03_12-46-09Z ##[debug]POLL URL RESULT: ***"statusCode":202,"statusMessage":"Accepted","headers":***"content-length":"695","connection":"close","content-type":"application/json; charset=utf-8","date":"Mon, 03 Jul 2023 13:00:31 GMT","server":"Kestrel","location":"https://***.scm.azurewebsites.net:443/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2023-07-03_12-46-09Z"***,"body":***"id":"cdd0ef27-1cc1-4c42-bdeb-fff3b2edcc0b","status":0,"status_text":"Building and Deploying 'cdd0ef27-1cc1-4c42-bdeb-fff3b2edcc0b'.","author_email":"N/A","author":"N/A","deployer":"GITHUB_ZIP_DEPLOY_FUNCTIONS_V1","message":"***\"type\":\"deployment\",\"sha\":\"27b3597e8c7f3a8242b9ca3cceb66284a22111bd\",\"repoName\":\"******/pms-***-az\",\"actor\":\"ne******\",\"slotName\":\"production\"***","progress":"Number of uids 1","received_time":"2023-06-29T09:25:44.2061665Z","start_time":"2023-06-29T09:25:47.4543441Z","end_time":null,"last_success_end_time":null,"complete":false,"active":false,"is_temp":false,"is_reado... ##[debug]Deployment status: 0 'Building and Deploying 'cdd0ef27-1cc1-4c42-bdeb-fff3b2edcc0b'.'. retry after 5 seconds

Answer 8 · 2023-07-17T15:03:18.000Z

This issue is idle because it has been open for 14 days with no activity.

Answer 9 · 2023-07-18T07:18:00.000Z

Same issue here, we have several Azure Function apps and at some point in time the deploy action starts to hang. Every consecutive run will timeout as well. Recreating the Function (using Pulumi) will allow us to deploy again, but after several days it starts to hang again :(

The debug log shows it's stuck at Updating submodules. with the timestamp of the 17th. Note that this deployments logs are from the 18th.

##[debug]POLL URL RESULT: 
{
  "statusCode": 202,
  "statusMessage": "Accepted",
  "headers": {
    "content-length": "675",
    "connection": "close",
    "content-type": "application/json; charset=utf-8",
    "date": "Tue, 18 Jul 2023 06:33:10 GMT",
    "server": "Kestrel",
    "location": "http://***.scm.azurewebsites.net:80/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2023-07-18_06-19-26Z"
  },
  "body": {
    "id": "ec7dfb32-94db-4356-b776-14787381bc44",
    "status": 0,
    "status_text": "",
    "author_email": "N/A",
    "author": "N/A",
    "deployer": "GITHUB_ZIP_DEPLOY_FUNCTIONS_V1",
    "message": "{\"type\":\"deployment\",\"sha\":\"a11623812a6f115ac269bdb396d2a57dd722f623\",\"repoName\":\"***\",\"actor\":\"***\",\"slotName\":\"production\"}",
    "progress": "Updating submodules.",
    "received_time": "2023-07-17T17:11:20.0270155Z",
    "start_time": "2023-07-17T17:11:20.0270155Z",
    "end_time": null,
    "last_success_end_time": null,
    "complete": false,
    "active": false,
    "is_temp": false,
    "is_readonly": true,
    "url": null,
    "log_url": null,
    "site_name": "***"
  }
}

Screenshot of azure portal shows the progress is not the latest reported progress.

Answer 10 · 2023-07-18T07:21:41.000Z

The log details, correlated to the handing github action, show that the function was actually deployed successfully.

(and there are no log details for the deployment that is marked as pending)

Answer 11 · 2023-07-18T07:28:42.000Z

If i delete all but the latest log record (aka, remove the logs for the pending deployment), the deploy action works again.

Answer 12 · 2023-08-01T09:03:17.000Z

This issue is idle because it has been open for 14 days with no activity.

Answer 13 · 2023-11-17T10:47:12.000Z

Still happpens...I deploy regularly across multiple function apps and then this starts happening for no obvious reason.

Answer 14 · 2023-11-17T16:33:14.000Z

I'm completely ignorant of the internals, but would it be possible to use Kudu Publish instead of Zip Deploy as is done in azure/webapps-deploy@v3? That solved a similar issue for me with deploying App Services.

Answer 15 · 2023-12-01T18:03:23.000Z

This issue is idle because it has been open for 14 days with no activity.

Answer 16 · 2024-02-09T19:57:40.000Z

If i delete all but the latest log record (aka, remove the logs for the pending deployment), the deploy action works again.

Hello,
I'm new to Azure Apps and devops. What do you mean by deleting logs ? Do you delete them from Deployment Center?

Edit: I deleted them from Deployment Center and the pipeline completed.

Answer 17 · 2024-02-16T08:05:45.000Z

If i delete all but the latest log record (aka, remove the logs for the pending deployment), the deploy action works again.

Hello, I'm new to Azure Apps and devops. What do you mean by deleting logs ? Do you delete them from Deployment Center?

Edit: I deleted them from Deployment Center and the pipeline completed.

I had the same issue regarding Azure Pipelines. But I do not use Github Actions but Azure Repos. Here deleting all but the latest logs from the Azure Deployment Center also solved the issue instantly.

Answer 18 · 2024-03-11T18:58:54.000Z

I'm having the same issue and can't implement the self-help fix of deleting logs from the Deployment Center. I'm getting a "Delete Failed" with no explanation. Any suggestions would be much appreciated! Also, @Azure team, please fix this.

Answer 19 · 2024-05-24T09:17:51.000Z

Also having the same problem (tried both azure/webapps-deploy@v2 and azure/webapps-deploy@v3). Github has been blowing all budgets with multiple 6-hour deployment runs. I have been able to mitigate to a degree by setting a timeout via timeout-minutes: X on the job like this:

[...]
deploy:
    needs: build
    name: Deploy
    runs-on: windows-latest
    timeout-minutes: 10
    steps:
[...]

So the job will get stuck and then be canceled after the timeout hits.

Re-running the CD job with debug logging enabled shows the infinite loop on this section of the log, which repeats until the timeout hits and the job gets killed, note the Deployment successful. message that apparently GitHub cannot parse correctly:

2024-05-24T08:27:38.8668130Z ##[debug]setting affinity cookie ["ARRAffinity=????????????0f3cd06517910600fe9f841d332f594d199be0459dd3c476329f;Path=/;HttpOnly;Secure;Domain=???????????????????.scm.azurewebsites.net","ARRAffinitySameSite=????????????0f3cd06517910600fe9f841d332f594d199be0459dd3c476329f;Path=/;HttpOnly;SameSite=None;Secure;Domain=???????????????????.scm.azurewebsites.net"]
2024-05-24T08:27:38.8673993Z ##[debug][GET] https://??????????????.scm.azurewebsites.net:443/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY&time=2024-05-24_08-18-40Z
2024-05-24T08:27:40.2531567Z ##[debug]POLL URL RESULT: {"statusCode":202,"statusMessage":"Accepted","headers":{"content-length":"827","content-type":"application/json; charset=utf-8","date":"Fri, 24 May 2024 08:27:39 GMT","server":"Kestrel","location":"https://??????????????.scm.azurewebsites.net:443/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY&time=2024-05-24_08-18-40Z"},"body":{"id":"????????-????-????-8591-1823d4ac83f5","status":0,"status_text":"Building and Deploying '????????-????-????-8591-1823d4ac83f5'.","author_email":"N/A","author":"N/A","deployer":"GITHUB_ZIP_DEPLOY","message":"{\"type\":\"deployment\",\"sha\":\"????1a897d95f31e829837bf67739be7fe0ea7ea\",\"repoName\":\"???????????/????????\",\"actor\":\"???????\",\"slotName\":\"production\",\"commitMessage\":\"???????????\"}","progress":"Deployment successful. deployer = GITHUB_ZIP_DEPLOY deploymentPath = ZipDeploy. Extract zip.","received_time":"2024-05-21T21:01:36.362926Z","start_time":"2024-05-21T21:01:37.9507836Z","end_time":null,"last_success_end_time":null,"complete":false,"active":false,"is_temp":false,"is_readonly":true,"url":null,"log_url":null,"site_name":"???????????????????","build_summary":{"errors":[],"warnings":[]}}}
2024-05-24T08:27:40.2542353Z ##[debug]Deployment status: 0 'Building and Deploying '????????-????-????-8591-1823d4ac83f5'.'. retry after 5 seconds

Using @v2 or @v3 does not seem to make a difference. I was kinda hoping that would fix it. The debug logs for both of them indicate the same loop.

Anyone have any ideas?

Answer 20 · 2024-05-29T12:24:01.000Z

Shot in the dark about a possible cause and solution, after I experienced the same issue as in the OP.

On my side this may have been caused by running a Bicep deployment to update some configs on the Function app. The Bicep deployment would remove the WEBSITE_RUN_FROM_PACKAGE = 1 from the FunctionApps AppSettings (Environment Variables). Removing WEBSITE_RUN_FROM_PACKAGE may have caused various issues when the app tried to restart after the config changes, and it seems that subsequent deployments would in some cases fail due to this corruption.

The fix was to re-add WEBSITE_RUN_FROM_PACKAGE=1 to the environment variables.

I'm still not sure if this is the real issue, but if this seems to correlate with other peoples experiences it would be nice to validate.

Answer 21 · 2024-07-30T09:35:29.000Z

I faced a similar issue after adding environment variables to my code.

I tried various solutions, including adding the .env file to my .gitignore, modifying my YAML configuration, and tweaking my requirements.txt file. However, the deployment process, which usually took 3-4 minutes, would not complete and had to be manually canceled each time.

Ultimately, I resolved the problem by deleting all logs from the deployment center on Azure since the initial deployment and redeploying the code. This approach finally worked!

Answer 22 · 2024-08-09T09:37:54.000Z

Adding WEBSITE_RUN_FROM_PACKAGE = 1 to environnent variables solved the issue. Deployements went from 10 min or more to 3 min!!

Answer 23 · 2024-09-09T17:35:01.000Z

Same here, I also checked the "Deployment slot setting" checkbox, no idea if it's necessary but it worked out. Deployment took less than 2 minutes.