The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Monday, September 10, 2007

Notes and stats on a graylist experiment

I started seeing replies to an email a friend sent (he sent it to a bunch of friends, who started replying to all) way before I got the original email my friend sent. When I checked, it was as I feared, a large company (Adelphia) had multiple machines for outoing mail, and each attempt was coming from a different IP address, and coming too quickly to pass through the embargo timeout. For a while, I was actually afraid it would never make it through. When I did finally get it, some 9½ hours had passed from the first attempt:

Sep 10 08:06:55 brevard graylist: tuple: [68.168.78.202 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 08:58:00 brevard graylist: tuple: [68.168.78.187 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 09:53:08 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 09:53:35 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 09:53:59 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 09:54:17 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 09:54:30 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 09:54:38 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 10:49:24 brevard graylist: tuple: [68.168.78.205 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 11:50:29 brevard graylist: tuple: [68.168.78.211 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 13:01:35 brevard graylist: tuple: [68.168.78.175 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:06:15 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:06:20 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:06:29 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:06:52 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:07:14 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:07:34 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:08:07 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:08:24 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:08:33 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 14:08:41 brevard graylist: tuple: [68.168.78.181 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 15:12:39 brevard graylist: tuple: [68.168.78.44 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:17:17 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:17:23 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:17:45 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:17:53 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:17:59 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:18:06 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 16:18:51 brevard graylist: tuple: [68.168.78.196 , XXXXXXXX@adelphia.net , sean@conman.org]
Sep 10 17:20:50 brevard graylist: tuple: [68.168.78.178 , XXXXXXXX@adelphia.net , sean@conman.org]

It's this behavior that has us at The Office concerned about greylisting; that delays of this magnitude will have our customers screaming at us. I've been keeping track of such emails, building up a list of IP addresses to immediately whitelist. P asked if the given IPs were listed as the MX record, and if so, use that to whitelist the email. But when I checked, that wasn't the case for Adelphia. P then suggested I check the SPF records.

Not a bad idea. The SPF record for Adelphia matched the IPs I was seeing. I then went on to check the SPF record for some of the other companies I was whitelisting, like AOL and BellSouth. Sure enough, most (Yahoo is the only exception so far) have SPF records. I may have to include an SPF check into the daemon, but I'd rather not immediately let through emails that pass the SPF check. I'll have to think about how I want to do this.

Meanwhile, some stats from the currently running version (started sometime last week):

Current Graylist statistics
tuples 1,810
graylisted 20,775
whitelisted 42
graylist expired 18,965
whitelist expired 0

The row labeled “tuples” are all the tuples currently in memory (that haven't expired) and the row labeled “graylisted” have been the number of emails added to the graylist since the program started. It's been holding steady at about 1,800 tuples at any one time for the past few days (and this is just the emails being sent to my server—perhaps a dozen domains or so, but mostly to conman.org). So far, only 0.2% of all emails have been whitelisted, but that includes 18 spams. Not that bad considering prior to this I was getting something like 1,800 per day.


I can see this as being as sharp as a sword

[Somehow, I get the feeling that this wouldn't be much of a match against a sword]

Despite some mistakes, the calligraphy I've done so far has been good enough that I was given a glass pen in appreciation.

Dubious about it actually being a pen, I tried it. It worked. It holds on to enough ink to get a few words down before needing to reload it. It's perhaps the coolest pen I've ever owned.

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.