httpstat.us service down?

Question

httpstat.us service down?

Closed this issue 3 years ago · 25 comments

every endpoint (including /404, /200, and /) yields 503 (observed between 7:56 and 8:46 am EDT)

Answer 1 · 2020-09-25T21:42:51.000Z

This is down for us as well and we're using it in our test suite for https://frontity.org, so I've created a dead simple app on glitch.me that we use instead. Works just fine 👌

Anybody having the same issue, you're welcome to use it too: https://ballistic-western-donkey.glitch.me 🙂

Answer 2 · 2020-09-27T22:06:16.000Z

Hey folks 👋

Thanks for reporting this.
I just tried a few endpoints, https://httpstat.us/404, https://httpstat.us/200, and they both worked.

Could you please confirm it's also working on your end?

Cheers

Answer 3 · 2020-09-28T03:11:02.000Z

Hey, @mderriey 👋

It does seem to work fine now 👍

Answer 4 · 2020-09-28T05:55:44.000Z

Thanks for confirming.

I'm closing this for now, feel free to reopen if it happens again in the near future.

Answer 5 · 2020-09-28T09:26:17.000Z

Hi, @mderriey

I'm afraid it seems to be down again. Right now, 11:25 am CET, it returns 503 or 524 (a Cloudflare timeout error).

Answer 6 · 2020-09-28T22:15:51.000Z

Thanks for letting us know, we'll put that on our todo list.

Answer 7 · 2020-09-28T22:56:46.000Z

Just an update, it seems that the service is getting absolutely slammed at the moment, averaging 3,000 requests per minute (peaking close to 7,500).

As a result, it's just not capable of meeting the demand, so we're trying to get to the bottom of that.

Answer 8 · 2020-10-05T10:22:07.000Z

Rate limiting (say, 60-100 requests per minute from a single IP?) would help in kicking out noisy clients. Dis you consider something like this?

Answer 9 · 2020-10-08T22:57:42.000Z

Hello! We use the site to validate some tests and our builds are failing due to the site being down. Going to the site times out.

Is the site still getting slammed with requests?

Answer 10 · 2020-10-08T23:20:27.000Z

Rate limiting (say, 60-100 requests per minute from a single IP?) would help in kicking out noisy clients. Dis you consider something like this?

We did, yes, however the tech stack as it stands is dated and we don't want to invest too much effort.
We're thinking about migrating to ASP.NET Core (see #91), which would make it much easier to introduce rate limiting.

Hello! We use the site to validate some tests and our builds are failing due to the site being down. Going to the site times out.

Is the site still getting slammed with requests?

I personally don't have access to the telemetry service. @aaronpowell?

Answer 11 · 2020-10-09T03:38:57.000Z

Short answer - yep!

I've tweaked some of the logging metrics to try and get a better handle on the sources to see if we can do some better limits.

Answer 12 · 2020-10-09T05:36:11.000Z

It looks like we're back up and running, so to speak.

TL;DR - in the last 24 hours we received 5.5m requests, of which I class 4.1m as abuse requests, and a temporary workaround is in place.

What happened?

At some point in the past, httpstat.us started receiving a lot of requests for one status code, a little over 52m requests in fact, and they appear to be setting the max-timeout for a request (10s).

Because of the way the timeout works, it deadlocks the thread until it expires, meaning there's only so many concurrent timeout requests that can be handled. Yeah, it's not the best solution, but as @mderriey said, it's a really dated tech stack and never designed to support that many requests.

Today, I pushed a small hack to basically noop the request to /504, which is the endpoint that is being abused, and as such, the service appears to be back to stable.

We'll keep monitoring for a few days to see if the abuse shifts, but at the very lease, it's holding together ok. 😂

Appologies to anyone who legitimately needs to hit /504, but for the moment, it's going to be offline.

Answer 13 · 2020-10-09T23:42:11.000Z

Looking at where we stand today, it's looking a lot better. Since the workaround went it, we're seeing an average response time where I expect it to be (sub 1s), rather than close to 10s when it was being abused.

It still looks like there's a lot of requests that I class as abusive requests, but they are no longer impacting the service.

I'll keep monitoring throughout next week and provide updates here.

Answer 14 · 2020-11-10T08:37:22.000Z

@aaronpowell our test case is using http://httpstat.us/504?sleep=3000 but now it doesn't work as expected
is this expected result according to your changes from Oct 9?

Answer 15 · 2020-11-10T08:58:05.000Z

Correct @helloint. At the moment we flat-out reject every request for a 504 response.

Answer 16 · 2020-11-11T03:22:40.000Z

well looks like it won't be fixed in short term. we will replace with http://httpstat.us/500?sleep=3000 then.
thanks.

Answer 17 · 2021-01-25T21:38:02.000Z

It looks like the service is down again, any idea what is causing it?

Answer 18 · 2021-01-25T22:27:28.000Z

Looks like there's an excessive traffic spike at the moment and azure is throttling the service. It's a public holiday here in Australia so I've only had about 5 minutes to look at it so far and probably can't get to it til later Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Thomas Schaper <notifications@github.com> Sent: Tuesday, January 26, 2021 8:38:17 AM To: Readify/httpstatus <httpstatus@noreply.github.com> Cc: Aaron Powell <me@aaron-powell.com>; Mention <mention@noreply.github.com> Subject: Re: [Readify/httpstatus] httpstat.us service down? (#89) It looks like the service is down again, any idea what is causing it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#89 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AADJ7XF26SWEF57SBA6WYXLS3XQETANCNFSM4RZQD5LQ>.

Answer 19 · 2021-01-25T22:37:46.000Z

Oh no worries, enjoy your holiday! It seems up again so I'll quickly restart our CI jobs :)

Thanks for your work!

Answer 20 · 2021-06-09T11:31:55.000Z

Service down as of now (June 9, 2021 at 13:30 CET). Returns 503 Service Unavailable.

Answer 21 · 2021-06-09T15:35:37.000Z

Also seeing that the service is down

Answer 22 · 2021-06-09T23:01:36.000Z

Looks like it righted itself. FWIW - these notifications came in at 9.30pm and 1.30am, neither of which times I was awake 🤣

Answer 23 · 2021-06-10T00:04:28.000Z

All good! I'm seeing it's working too. Cheers

Answer 24 · 2021-12-20T23:53:21.000Z

I'm finally closing this. I've done some work on the Azure infra, it's been ported off .NET Framework to .NET 6 which sees better performance. I've also changed the SSL handling to be Azure directly, so we shouldn't see the certificate expire.

I'm still observing some abuse around the sleep and 504 requests, so they'll be left as a no-op for the time being.

Answer 25 · 2023-12-12T18:47:13.000Z

Hi there, first of all, this service is really cool - thanks for offering it! I did want to let you know that I'm currently seeing that the site is (mostly?) down:

$ curl -v https://httpstat.us/200
*   Trying 20.40.202.3:443...
* Connected to httpstat.us (20.40.202.3) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=httpstat.us
*  start date: Nov  7 00:00:00 2023 GMT
*  expire date: May  7 23:59:59 2024 GMT
*  subjectAltName: host "httpstat.us" matched cert's "httpstat.us"
*  issuer: C=US; O=DigiCert, Inc.; CN=GeoTrust Global TLS RSA4096 SHA256 2022 CA1
*  SSL certificate verify ok.
* using HTTP/1.1
> GET /200 HTTP/1.1
> Host: httpstat.us
> User-Agent: curl/7.88.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 503 Service Unavailable
< Content-Length: 0
< Date: Tue, 12 Dec 2023 18:45:31 GMT
< Set-Cookie: ARRAffinity=2f58bd0c4abeddfa81d72684f97e0104b7cea9d1afb9a5c44c71bf44646c500b;Path=/;HttpOnly;Secure;Domain=httpstat.us
< Set-Cookie: ARRAffinitySameSite=2f58bd0c4abeddfa81d72684f97e0104b7cea9d1afb9a5c44c71bf44646c500b;Path=/;HttpOnly;SameSite=None;Secure;Domain=httpstat.us
<
* Connection #0 to host httpstat.us left intact