Monday, September 27, 2004
The evergrowing spam problem
While I was staying at Hotel Negiyo, I had the chance to talk with Rob, my old roommate—he's a sysadmin there at Negiyo. We chatted for a bit about the server compromises I experienced then drifted towards the new anti-spam measures he helped to implement at Negiyo.
Quite impressive actually. The router sitting in front of the email system is actually one of two that are synchronized; if one goes down the other can immediately take over without dropping an existing TCP connection. The network traffic then goes to one of several load balancers which send the traffic through one of fourty dedicated systems that do nothing but scan the email, looking for signs of spam. It's more of a filter, as these systems then send the traffic on towards the actual email server for actual delivery. As the email filters through the dedicated system(s), it makes a determination if the email is spam or not, and updates a database system with a yea/nay flag that the SMTP server can then query and deal with the spam as the customer would like.
The volume of email they handle just in Boca is staggering—Rob showed me the statistics for the previous day: 15 million pieces of email. Even more impressive (and depressing) is that out of 15 million pieces of email, more than 80% was classified as spam.
Eighty percent!
Out of 15 million emails delivered, over 12 million was nothing more than spam.
Talk about depressing.
