The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Monday, November 26, 2007

Musings on high volume email servers and X-Grey, the greylist daemon

On Saturday, I bumped into Rob at a “After Thanksgiving Party” and we discussed the use of X-Grey at Negiyo, at least, those parts of Negiyo email that Rob helps to manage.

The code, as is, won't work with their setup. First problem, the sheer volume of email—something like 100,000 connections per second. These are fed through two load balancers and farmed out to about 100 servers, so each server is responsible for 10,000 connections per second. While I suspect X-Grey can handle 10,000 connections per second, the major problem are the load balancers—there's just no guarantee that the load balancers will be consistent on which machine they send the connection to.

For instance, we have some machine, on IP address 10.20.30.40 sending an email from alice@example.net to bob@negiyo.com. The load balancer will send that to server A, which doesn't find the tuple [10.20.30.40 , alice@example.net , bob@negiyo.com], stores it for later reference, and sends back “try again later.” Later, the machine at 10.20.30.40 tries sending the email again, only this time, the load balancer sends the connection to server B, which doesn't find the tuple, stores it, and sends back “try again later.” Lather, rinse, repeat until the sender gives up, or the load balancer manages to send the traffic to a machine that actually has the tuple stored.

There's just no way of knowing which server the load balancer will send the traffic to. So, we point all the servers to a single greylist server, which now has to handle 100,000 requests per second. Okay, so assuming X-Grey can handle that load (it's a real beefy box on a fat pipe), and given that we store greylisted tuples for six hours … carry the one … 2,160,000,000 tuples.

Blink.

Blink.

Okay, now that I'm actually doing the math instead of sitting around in a comfortable chair listening to Rob while chowing down on turkey and stuffing, I find it rather difficult to believe that Negiyo is getting around 8½ billion emails per day—even a billion per day is stretching my credibility. The worst we get at The Company is 8 per second, with an average hovering around 1.4 (or 122,540 per day, which I calculated twice, using two different statistics that are recorded). More believable is 100,000 per hour (or even up to 1,000,000 per hour, which is 11 emails per second).

I'll have to get back with Rob on this …

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.