The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Thursday, Debtember 01, 2005

Yet more thoughts on spam

Last month I switched email clients from Thunderbird to mutt (I found Thunderbird to be too sluggish but that's a story for another entry) and configured our primary email server to forward my mail directly to my workstation, where procmail can then filter it.

So now I can burn through mail in about half the time it used to take me.

I get a ton of email, most of it from the various servers (from root mostly) and most of that is generated by the mail system itself, informing me that it's found, yet again, another email infected with a virus (oh, easily 500 a day) or it couldn't deliver a message (another 500 a day easy) or the multi-thousand line output of logwatch (each easily 15,000 lines of summary per day).

So it was a simple matter to set up procmail to filter the messages (and say, automatically delete the virus warnings—I tried turning that off on the servers themselves, but … well … control panels and hidden configuration files and I'm stuck getting them even though I don't care for them). Now, since our mail goes through a dedicated spam filtering system and can mark emails as spam, I thought it would be a good idea to simply delete those upon receipt as well.

Only I kept receiving emails marked as spam.

  31 N   Dec 01 trespassers@gre ( 306) [SPAM]  Breaking News

Puzzled, I moved the procmail configuration to delete such marked spam:

:0:
	* ^Subject: .*SPAM.*
	in-TRASH

to the start of my .procmailrc, and yet, I still get the emails. I bumped up the verbosity of logging, and yes, some of it was actually being caught and trashed, but not all of it.

What the heck?

In mutt I see:

From: <trespassers@greenoblivion.com>
To: <apache@XXXXXXXXXXX>
Subject: [SPAM] Breaking News
Date: Thu, 1 Dec 2005 22:49:10 +0200

But when I checked the actual raw email message …

From: <trespassers@greenoblivion.com>
To: <apache@XXXXXXXXXXX>
Subject: =?ascii?B?W1NQQU1dICBCcmVha2luZyBOZXdz?=
Date: Thu, 1 Dec 2005 22:49:10 +0200

That funky subject line? A form of MIME encoding for email headers. In this case, the subject line uses the US-ASCII character set and is encoded as base-64. procmail knows nothing about MIME encodings. It's looking for “SPAM” in the subject line and not finding it.

Well now …

Obviously, I can add

:0:
	* ^Subject: =\?.*\?W1NQQU1dIC.*
	in-TRASH

(“[SPAM]” encoded as base-64) to my .procmailrc file, but is there a better way?

Sure, Bayesian filtering is pretty cool, but I still think that a few simple heuristics in place would help just as much.

One idea: check the character encoding of the incoming email. In my case, if it isn't US-ASCII, ISO-8859-1 or UTF-8 (oh, might as well include WINDOWS-1251 for those unfortunate friends that are abused by Microsoft), then discard it. It doesn't matter if it's legitimate email if I don't understand the language it's written in.

Now, with ISO-8859-1, UTF-8 or WINDOWS-1251, I still might not be able to read the message (since ISO-8859-1 and WINDOWS-1251 covers western European langauges like French and German, and UFT-8 covers just about all written languages), but my second idea should take care of that.

Second idea: spell check the incoming email.

No, seriously.

Take this bit of spam I received today:

lt is really hard to recollect a company: the market is full of sugqestions and the information is overwhelming; but A GOOD CATCHY LOGO, STYLISH STATlONERY and OUTSTANDING WEBSIT E wilI make the task much easier.

We do not promise that having ordered a loqo your company wiIl automaticaIly become a worId Ieader: it is quite clear that without good products ,effective business orqanization and practicable aim it will be hot at nowadays market; but we do promise that your marketing efforts will become much more effective.

Twelve spelling errors (and one punctuation error, which I marked, but not counting in the following statistic) for a 14% spelling error rate. And if the email is in a different language, the spelling error rate will easily go past 95%. So, if the number of misspelled words exceeds say, 70%, delete it, and if it's above say, 5% (hey, we all make mistakes sometimes) mark it as possible spam.

This would definitely piss off the V1@gr@ pushers.

Third idea: Unless whitelisted, any email that consists of any type of attachment, delete it (well, for me at least).

And this is before explicit filtering, Bayesian or otherwise.

I wonder just how hard something like that would be to write …

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2019 by Sean Conner. All Rights Reserved.