Investigation in Burner.Kiwi
Closed this issue · 3 comments
Hi all,
Sorry, this isn't an issue as such, just somewhere to open a discussion.
I currently own a temporary email service similar to that of Burner.Kiwi, although mine is written in PHP and relies heavily on IMAP. While this has worked reasonably well until now, the IMAP server is struggling to keep up with the number of concurrent connections to the single inbox so I am trying to find an alternative. Burner.Kiwi looks quite promising so far.
Firstly, can I ask, how many concurrent users could Burner.Kiwi handle using SQLite, or is this entirely limited by server resource? If this could handle say a million emails a day and 1,000 concurrent users, then I might go ahead and do this.
Ideally I would like to run Burner.Kiwi in a server-less environment, but this would require MailGun, which is expensive. I see someone has talked about using SendGrid before which is a lot cheaper although this isn't built. Do you know if anyone has started such a thing? I might be interested in building this, although I might need a little help.
Apart from that, I am trying to think of alternative ways which Burner.Kiwi could handle incoming emails in a server-less environment. I am wondering if it would be possible to use AWA SES somehow? From my initial investigation emails could be processed by AWS SES and put into an S3 container, or, piped into a Lambda function. From there anything can happen, maybe onwards to MySQL, Postgres, DynamoDB? Likewise, an Azure LogicApps could also automatically pick up emails from an Outlook inbox and parse that into a database.
Any thoughts?
Sorry, this isn't an issue as such, just somewhere to open a discussion.
No worries. More than happy to have this discussion.
Firstly, can I ask, how many concurrent users could Burner.Kiwi handle using SQLite, or is this entirely limited by server resource? If this could handle say a million emails a day and 1,000 concurrent users, then I might go ahead and do this.
The answer is, I'm not sure. I've never actually bench marked this project to see what it can handle and I don't have any performance monitoring/analytics in place in "production". The current sqlite connection parameters need a bit of tuning but the common wisdom I see online is that sqlite and go can handle a lot of traffic with the correct params. Obviously, it would ultimately depend on what resources you're giving it. I think you'd need to run some benchmarks with the configuration you would like to fully answer that question.
Could any configuration of burner.kiwi handle this much traffic? Absolutely. I have no doubt you could horizontally scale this project to reach whatever traffic level you need.
Ideally I would like to run Burner.Kiwi in a server-less environment, but this would require MailGun, which is expensive. I see someone has talked about using SendGrid before which is a lot cheaper although this isn't built. Do you know if anyone has started such a thing? I might be interested in building this, although I might need a little help.
I would recommend against using serverless if you're going to be providing this sort of service for free to the internet. That was what I initially used when I first built this project, however, someone registered and sent thousands of emails to burner.kiwi. Serverless, of course, scaled to handle this and charged me several hundred dollars for what amounted to less than an hours worth of wall time.
As for sendgrid, it has not been implemented to my knowledge. This should be relatively simple to implement and I'm more than happy to point you in the right direction, do code review and merge it into this repo.
Also FYI if you weren't aware, burner.kiwi does have a built in SMTP server. It can receive it's own email without the need for an external provider. But that won't work in some serverless environments.
Apart from that, I am trying to think of alternative ways which Burner.Kiwi could handle incoming emails in a server-less environment. I am wondering if it would be possible to use AWA SES somehow? From my initial investigation emails could be processed by AWS SES and put into an S3 container, or, piped into a Lambda function. From there anything can happen, maybe onwards to MySQL, Postgres, DynamoDB? Likewise, an Azure LogicApps could also automatically pick up emails from an Outlook inbox and parse that into a database.
I'm sure that there are many ways you could do this in a serverless way. However, as I said above, I'd recommend against it to protect your wallet and just from a complexity perspective. It's also not a direction I want to take the project in, I've even been thinking about removing AWS lambda support because I don't use it and can't guarantee it works correctly. I'd be interested to see where you'd take it but you're on your own here.
Also more generally if you do truly handle that much traffic then burner.kiwi needs some other work. Off the top of my head it would need better logging and performance monitoring, work on the SMTP email parser to ensure that it handles most of the emails people throw at it, as well as some performance tuning.
Happy to keep discussing and help out where I can.
Yes, this is my concern with using MailGun, I'd essentially be paying for someone else's spam which 99% of won't get read. The SMTP is certainly cheaper especially considering the prices of VPS's in Germany (4 vCPU, 8 GB RAM, 32 TB Traffic, 50 GB NVMe for just €4.99).
While it's nice to have everything in one app, do you think there would be any benefit to splitting up the user-interface, and the SMTP back-end? That way you could have multiple SMTP servers, potentially load-balancing between them. In this case the user-interface app would read emails from either MySQL, Postgres, MailGun, SendGrid, SQLite, DynamoDB, maybe even some other options too like Firebase. The app could be hosted, or server-less. While the SMTP is a separate app which can pipe emails into the above databases. This would also allow people to pipe their own emails using a custom app, PipeDream, AWS step functions, Azure LogicApps, etc...
Another consideration is my site is protected by CloudFlare to prevent against DDOS and so that I can enforce rate limiting. The problem is MX records aren't proxied, and thus expose the IP address of the server. At least having them split, the user-interface would be protected.
While it's nice to have everything in one app, do you think there would be any benefit to splitting up the user-interface, and the SMTP back-end?
Yep that's definitely a possibility. I think you would have to be careful of premature optimization and only do it when it makes sense. You might be able to get very far with a single instance and a single database.
You might want to check out https://github.com/flashmob/go-guerrilla. It has a database interface and it could be easier to go down that route. Could even use firebase database and a js frontend to sync things without having to write syncing code. Just a thought.
Another consideration is my site is protected by CloudFlare to prevent against DDOS and so that I can enforce rate limiting.
I think Cloudflare Tunnels could be your answer here. It looks like they support tunneling arbitrary tcp ports.