Mailinator Architecture
Update: A fun exploration of applied searching in How to search for the word "pen1s" in 185 emails every second. When indexOf doesn't cut it you just trie harder.
Has a drunken friend ever inspired you to create a first of its kind internet service that is loved by millions, deemed subversive by thousands, all while handling over 1.2 billion emails a year on one rickity old server? That's how Paul Tyma came to build Mailinator.
Mailinator is a free no-setup web service for thwarting evil spammers by creating throw-away registration email addresses. If you don't give web sites you real email address they can't spam you. They spam Mailinator instead :-)
I love design with a point-of-view and Mailinator has a big giant harry one: performance first, second, and last. Why? Because Mailinator is free and that allows Paul to showcase his different perspective on design. While competitors buy big Iron to handle load, Paul uses a big idea instead: pick the right problem and create a design to fit the problem. No more. No less. The result is a perfect system architecture sonnet, beauty within the constraints of form.
How does Mailinator carry out its work as a spam busting super hero?
Site: http://mailinator.com/
Information Sources
The Platform
The Stats
The Architecture
- Design a system that values survival above all else, even users. Survival is key because Mailinator must fight off attacks on a daily basis.
- Provide 99.99% uptime and accuracy for users. Higher uptime goals would be impractical and costly. And since the service is free this is just part of rules of the game for users.
- Support the following service model: user signs-up for something, goes to Mailinator, clicks on the subscription link, and forgets about it. This means email doesn't have to be stored persistently on disk. Email can reside in RAM because it is temporary (3-4 hours). If you want a real mailbox then use another service.
- Sendmail received email in a single on-disk mailbox.
- The Java based Mailinator grabbed emails using IMAP and/or POP (it changed over time) and deleted them.
- The system then loaded all emails into memory and let them sit there.
- The oldest email was pushed out once the 20,000 in memory limit was reached.
- It was stable and stayed up for months at a time.
- It used almost all the 1GB of RAM.
- Problems started when the incoming email rate started surpassing 800,000 a day. The system broke down because of disk contention between Mailinator and the email subsystem.
- The idea was to remove the path through the disk which was accomplished with a complete system rewrite.
- The web application, the email server, and all email storage run in one JVM.
- Sendmail was replaced with a custom built SMTP server. Because of the nature of Mailinator a full SMTP server was not necessary. Mailinator does not need to send email. And it's primary duty is to accept or reject email as fast as possible. This is the downside of layering. Layering is very often given as a key strategy in scaling, but it can kill performance because crucial decisions are best handled at the highest levels of the stack. So work flows through the system only to be dumped at the lower layers when many of the RAM and cycle stealing operations have already been accomplished. So the decision to go with a custome SMTP server is an interesting and brave decision. Most people at this point would just add more hardware. And they wouldn't be wrong, but it's interesting to see this path taken as well. Maybe with more DOM and AOP like architectures we can flatten the stack and get better performance when needed.
- Now Mailinator receives an email directly, parses it, and stores it into memory. The disk is bypassed completely and the disk remains fairly idle.
- Emails are written to disk when the system is coming down so they can be reloaded on startup.
- Logging was shut-off to remove the risk of subpoenaes. When logging was performed log data was written in batches so several thousand logs lines would be written in one disk write. This minimized at disk contention at the risk of losing helpful diagnostic information.
- The system uses under 300 threads. More aren't needed.
- On arrival each email passes through a filter system and is stored in RAM if all filters are passed.
- Every inbox is limited to only 10 emails so popular inboxes, like joe@mailinator.com, can't blow the system.
- No incoming email can be over 100k and all attachments are immediately discarded. This saves on RAM.
- Since 99% of emails are never looked at, compressed email saves RAM. They are only
ever decompressed when someone looks at them.
- Mailinator can store about 80,000 emails in RAM, using under 300MB of RAM compared to the 20,000 emails which were stored in 1GB RAM in the original design.
- With this pool the average email lifespan is about 3-4 hours.
- It's likely 200,000 emails could fit in memory, but there hasn't been a real need.
- This is one of the design details I love because it's based on real application usage patterns. RAM is precious and CPU is not, so use compression to save RAM at the expense of CPU, knowing you won't have to take the CPU hit twice, most of the time.
- There is no privacy. Anyone can read any inbox at anytime.
- Relaxing these constrains, while shocking, makes the design much simpler.
- For the user it is simple because there is no sign up needed. When a web site asks you for an email address you can just enter an mailinator address. You don't need to create a separate account. Typing in the email address effectively creates the mailinator account. Simple.
- In practice users still get a high level of privacy.
- Mailinator doesn't have anything against SPAM, but because it gets so much SPAM, it must be filtered out when it threatens the up time of the system.
- Which leads to this rule: If you do anything (spammer or not) that starts affecting the system - your emails will be refused and you may be locked out.
- Bounce: all bounced emails are dropped.
- IP: too much email from a single IP are dropped
- Subject: too much email on the same subject is dropped
- Potty: subjects containing words that indicate hate or crimes or just downright nastiness are dropped.
- An AgingHashmap is used to filter out spammers from a particular IP address. When an email arrives on a IP address the IP is put in the map and a counter is increased for all subsequent emails.
- After a certain period of time with no emails the counter is cleared.
- When a sender reaches a threshold email count the sender is blocked. This prevents a sender from flooding the system.
- Many systems use this sort of logic to protect all sorts of resources, like comments. You can use memcached for the same purpose in a distributed system.
- Spam can be sent from a large coordinates sets of different IP addresses, called zombie networks. The same message is sent from thousands of different IP addresses so the techniques for stopping email from a single IP address are not sufficient.
- This filtering is a little more complex than IP blocking because you have to parse enough
of the email to get the subject line and matching subject strings is a little more resource intensive.
- When something like 20 emails with the same subject within 2 minutes, all emails with that subject are then banned for 1 hour.
- Interestingly, subjects are not banned forever because that would mean Mailinator would have to track subjects forever and the system design is inherently transient. This is pretty clever I think. At the cost of a few "bad" emails getting through the system is much simpler because no persistent list must be managed and that list surely would become a bottleneck. A system with more stringent SPAM filtering goals would have to create a much more complex and less robust architecture.
- Nealy 9% of emails are blocked with this filter.
- From my reading Mailinator filters only on IP and subject, so it doesn't have to read the body of the email body to accept or reject the email. This minimizes resource usage when most email will be rejected.
- All connections that are silent for a specific period of time are droped.
- Mailinator sends replies to email senders very slowly, like 10 or 20 or 30 seconds, even for a very small amount of data. This slows down spammers who are trying to send out spam as fast as possible and may make them rethink sending email again to that address. The wait period is reduced during busy periods so email isn't dropped.
Lessons Learned
private accounts for everyone. But you really need these things? What can you toss?
when trying to conserve RAM. I've seen memory usage drop by more than half when using compression with very little overhead. If you are communicating locally, just have the client encode the data and keep it encoded. Build APIs to access the data without have to decode the full message.
those like Mailinator?
