Over quota error
czgdp1807 opened this issue · 14 comments
The website, live.sympy.org is facing denial of service attack from a past few days.
I think we need to increase the upper limit on the number of requests for the website. Or a protection is needed against such attacks.
May be we can refer, https://cloud.google.com/appengine/docs/standard/python/creating-firewalls
It happens at the exact same time every evening, for the past three days, from 5 PM to 1 AM Mountain Standard Time. I forgot to look at it yesterday.
It looks like blocking IPs with the firewall is straightforward. The thing I'm not able to figure out easily is which IP(s) are the offending ones. It should be easier to detect this once it is going on. I've set a reminder for today to look at it so hopefully we can catch it as it happens and determine in real time if the block works. If anyone has experience with reading Google Cloud logs, let me know.
Blocking a particular IP can be a solution. Since, it is happening at exactly the same time so it looks like someone has set up an automated script and has enough computational resources to perform this DoS attack. I am worried they may use VPN to change their IP and once again do the same.
Is it possible to obtain protection against such attacks through Google cloud? Do they provide it for free?
Is it a number of requests, or is it that they found some slow running function in SymPy that they are triggering? Is it a DoS attack on purpose, or does somebody have some script that uses live.sympy.org and just happens to go over quota?
It happened again today. From the logs, there seems to be a single IP that is making all the requests. It is the same IP as it was yesterday. Based on a geolocation, the IP is in London, England.
I blocked the offending IP, and the requests are already going down. Unfortunately, I think we will still be over quote for the rest of the day. I couldn't find any information on when the spending limit day resets. We can increase the daily spending limit (currently $8 a day) if necessary.
The good news is that the docs site still loads even with SymPy live being down. I was worried that the extension would block the page load, but it just loads without any SymPy live popup or buttons on the examples.
The patterns suggest that it was an automated DoS attack. Let's see if that happens again on blocking the suspect IP. If nothing similar happens again then we shouldn't take any further steps, otherwise we can look for some protection against DoS. Increasing requests is a temporary solution, at the end we should aim for detecting such attacks and stoping them automatically.
A simple and zero cost solution can be is to use Captcha before opening the SymPy live page, to stop automated requests to the page.
On receiving any request first we should load Captcha test and if the user passes the test then we should load the original page. If they fail then it's a robot and we should block the IP.
I think the code for live.sympy.org is based on django. If it's django based then it should be quite easy to integrate Google's Captcha test into it.
It is happening again,
Doing something like this is also a possibility, https://simpleisbetterthancomplex.com/tutorial/2017/02/21/how-to-add-recaptcha-to-django-site.html
Please let me know of the possibility of this idea.
@czgdp1807 as I noted, once we are over quota, we have to wait until it resets. The site appears to be up now.
The IP ban seems to have worked. The requests went down right away after I implemented it. We'll need to keep an eye on this to see if it happens again.
Implementing something more complicated like a DDoS protection or a captcha is a last resort. There's no need to even consider those things at this point, because as far as we know banning IPs works. Also, I don't believe a captcha wouldn't actually solve the problem unless the App Engine supports it directly. Spamming requests to a captcha and spamming requests to the site would both fill the quota.
It hasn't happened again in the past week. I am going to close this issue. If the site goes down again, open a new issue for it.