Question: recommendations for dealing with potentially malicious requests?
jafd opened this issue · 3 comments
As far as I understand it, Mummy is designed such that a handler gets invoked when a request is fully received and parsed. This, however, opens some possibilities of sabotaging the server by:
- sending requests so slowly that the worker threads get hogged up
- sending requests so large that they exhaust memory when a sufficient number of them is being received
- trying to hog up or crash the request parser by sending specially crafted requests
- any combination of the above
And there's also a use case of legitimately large request bodies and long running requests.
While I understand that some of these concerns may be addressed on the reverse proxy side, I don't think all of them can be dealt with there. Is there a way to hook into the request receiving process to terminate the connection early if bogus input is detected, even if not the whole input has been received? Is there a way to expose some iterator-like or file-like interface so that requests which are supposed to be large or containing large file uploads (or streaming ones, via chunked transfer mode) could be handled in a memory-efficient manner?
Regarding the possible malicious request scenarios:
Sending requests so slowly that the worker threads get hogged up
As explained in the readme, Mummy is not susceptible to low-and-slow. Only fully received requests are passed on to a worker thread so there is never any waiting on a client from a worker thread. Receiving from a client is an epoll loop that does not care how slow a client sends data (same for sending a response to a client).
Sending requests so large that they exhaust memory when a sufficient number of them is being received
In the end this is unavoidable (millions of small requests do the same thing). Ensure you have an appropriate max body size limit set, ensure you have some redundant capacity etc. In the event of an actual attack, mass numbers will take down load balancers, reverse proxies, and infrastructure along the way so thinking the API server can solve this on its own is not correct imo. If you can make a specific suggestion of how Mummy can improve in a specific way I'm of course open to it.
Trying to hog up or crash the request parser by sending specially crafted requests
Certainly this is possible, this would inevitably require code changes to address.
Is there a way to hook into the request receiving process to terminate the connection early if bogus input is detected, even if not the whole input has been received?
Not presently, and I suggest this needs a very specific example to be discussed more productively. Most bogus input (bad http, etc) result in the connection being dropped already.
Is there a way to expose some iterator-like or file-like interface so that requests which are supposed to be large or containing large file uploads (or streaming ones, via chunked transfer mode) could be handled in a memory-efficient manner?
Yes it is possible, but not something I am going to work in the foreseeable future. I specifically explain that this is not what Mummy is mode to do, while it certainly can of course work as it is just a matter of configuration. I suggest digging into object stores (like S3, R2, B2 etc) and pre-signed URLs. This will be far more cost-effective and far simpler. Simply provide clients with pre-signed upload URLs for approved large file uploads. They can ping to let you know the file is uploaded and then you can post-process it however you'd like. Alternatively NGINX has a configuration for file upload endpoints as well, utilizing a custom header and temp files IIRC. There are lots of options.
In other words, if I wanted to make a fork to make these things possible (specifically streaming requests/responses handling with an option to terminate a request early because whatever) for my own needs, are you saying that the chances that it would clash with your own plans to implement something similar would be low? That's nice if so.
That is correct, I think exploring these ideas with a fork is great. If you end up with something that is working well for you, consider letting me know if you're open to discussing the choices etc.
I don't foresee making significant changes that would make keeping a fork up to date with upstream being extra difficult.