PowerShell Gallery Extremely Slow Publishing and Access

Question

PowerShell Gallery Extremely Slow Publishing and Access

Opened this issue a year ago · 24 comments

Prerequisites

Write a descriptive title.
Make sure you are able to repro it on the latest version
Search the existing issues.

Steps to reproduce

Starting within the last 24 hours PowerShell Gallery has become extremely slow. I do not see anything on the status page yet but there should be.

Accessing the gallery is slow.
PSResourceGet is failing with: "Message": "Response status code does not indicate success: 503 (Service Unavailable). on and off. Example URL: "https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AWS.Tools.ECRPublic'&$inlinecount=allpages&$filter=IsLatestVersion%20and%20Id%20eq%20'AWS.Tools.ECRPublic'"
Publishing packages is extremely slow. It has taken about 2 hrs to publish 190 packages. Still has to publish 180 more packages where yesterday it took about 1 hr to finish (Normal). It is going to take it more than 4 hours to publish at this rate.

Expected behavior

Working with normal speed.

Actual behavior

It is extremely slow.

Error details

No response

Environment data

N/A

Version

N/A

Visuals

No response

Answer 1 · 2024-05-16T02:19:35.000Z

@boblodgett Thanks for letting us know, we've been doing some deployments which may have caused a bit of latency, though it shouldn't be that extreme. Do you know when exactly this started happening or just that it was roughly 24 hours ago? Things should be stable in terms of deployments, but please keep us updated if this issue continues or resolves. We'll investigate and see if we can pinpoint what's going.

Answer 2 · 2024-05-16T03:00:53.000Z

@alerickson our metrics show that the slowness with FindPackagesById() API call started around 2024-05-15 17:10 UTC and was about 6 times slower than the baseline and then started to get significantly better from 2024-05-16 00:40 UTC. Its still not as good as it used to be, about 40% slower than the baseline.

Answer 3 · 2024-05-16T16:16:16.000Z

@alerickson appears to be closer to baseline from 2024-05-16 15:10 UTC

Answer 4 · 2024-05-20T14:48:29.000Z

...we've been doing some deployments which may have caused a bit of latency,...

@alerickson how much latency was added to the system? Should we be seeing job steps that are going from 12 minutes up to 60+ minutes for downloading modules to our agents in Azure DevOps Service (use Azure VMSS deployed agents so our steps to install dependent modules has to run in every run).

Answer 5 · 2024-05-20T14:59:19.000Z

@alerickson - All of our builds are now failing/timing out when we use Invoke-PSDepend. Something is now broken.....

Answer 6 · 2024-05-20T15:04:23.000Z

@alerickson @SteveL-MSFT We are unable to publish modules at the moment. Are you folks working on a fix?

Answer 7 · 2024-05-20T18:05:53.000Z

@SydneyhSmith - FYI - serious performance issues with Powershell gallery again. Has been this way since late last week.

Answer 8 · 2024-05-20T18:24:32.000Z

Making some changes this morning and monitoring-- hopefully should be improving shortly

Answer 9 · 2024-05-20T21:28:16.000Z

Discovered this post after noticing that PowerShell modules which used to update in seconds, are now taking 5-30+ minutes. Still haven't seen Microsoft.Graph finish installing. Perhaps a load balancing, server I/O, or software timeout issue on the server side? Good luck with the fix!

Answer 10 · 2024-05-21T13:29:28.000Z

Performance is still very bad - 1 day later.

Answer 11 · 2024-05-21T13:54:30.000Z

Same for us. It was "OK" briefly in the morning (UK time) and then it went super slow again.

Answer 12 · 2024-05-21T15:21:39.000Z

I have been trying to install Azure Powershell for over 24 hours with no success.

Answer 13 · 2024-05-21T15:24:32.000Z

our pipelines have been experiencing the slowness (causing failures) since about Noon Central Time yesterday. It seems to happen in waves, and will occur for an hour or two, and then things work, and then later another hour or two of failures.

Answer 14 · 2024-05-21T16:50:53.000Z

Thanks for the continued updates, our monitors are showing that the Gallery is healthy at this point, after we have made updates, but we are doing deeper investigation based on these reports

Answer 15 · 2024-05-21T16:59:08.000Z

I've had a ticket open with the graph team since yesterday. It was recommended that I drop the ticket number here -

2*0007771

We've had pipelines fail with End of Central Directory could not be found. We pretty much build everything on demand for our infrastructure and deployments so we're completely down when these things timeout.

It seems like its a bit better today but there is still some huge delays. Any updates on what you're seeing on your end?

Answer 16 · 2024-05-21T17:05:41.000Z

Our monitors show it is still working very slowly. These problems started shortly before I opened this ticket and have not completely recovered since.

The requests are timing out for us:

https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AWS.Tools.AWSSupport'&$inlinecount=allpages&$filter=IsLatestVersion%20and%20Id%20eq%20'AWS.Tools.AWSSupport'. Attempt 1 of 3. 
{
    "Url": "https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AWS.Tools.AWSSupport'&$inlinecount=allpages&$filter=IsLatestVersion%20and%20Id%20eq%20'AWS.Tools.AWSSupport'",
    "Retry": 1,
    "MaxRetries": 3,
    "Exception": {
        "ClassName": "System.Threading.Tasks.TaskCanceledException",
        "Message": "The request was canceled due to the configured HttpClient.Timeout of 30 seconds elapsing.",

Answer 17 · 2024-05-21T17:36:33.000Z

We are also seeing issues across all of our environments with all different types of modules. Sometimes gets an error, sometimes it never completes and is just suspended. Have not seen this issue before

Answer 18 · 2024-05-21T18:55:56.000Z

We just finished another deployment-- let me know if you are seeing any improvement

Answer 19 · 2024-05-21T19:19:00.000Z

What were the causes of the issues?

Answer 20 · 2024-05-21T19:26:51.000Z

The initial cause points to high load on the service--- we are still determining the full series of events as to why the service did not scale as expected

Answer 21 · 2024-05-21T20:15:23.000Z

The initial cause points to high load on the service--- we are still determining the full series of events as to why the service did not scale as expected

It does seem a lot more performant right now.

Install of Az module was much faster.

Now testing install of Microsoft.Graph module which as well as being slow was also giving Package failed to install because: End of Central Directory record could not be found. earlier.

Answer 22 · 2024-05-21T20:22:09.000Z

Yeah definitely a lot better!


VERBOSE: Module 'Microsoft.Graph' was installed successfully to path '/home/prometheus/.local/share/powershell/Modules/Microsoft.Graph/2.19.0'.
PS /home/prometheus>

Answer 23 · 2024-05-21T20:32:20.000Z

It has currently improved to normal levels:

Answer 24 · 2024-05-23T22:04:55.000Z

Performance is much better on my end is well.