Tuesday, November 21, 2006
Perhaps Spring is right, we should tell our customers to use Gmail because we no longer support email
So much for a quiet day at the office.
A little background: a while ago I set things up so that all the email
that the various programs send out, on all the servers we have, gets funneled
into a single account, which is then forwarded to me for review: output from
cron, the mailscanner software, bounces, etc.
Yes, it generates a large amount of email, but mostly it's stuff I can
quickly scan and dump as all I'm really doing is looking for anything out of
the ordinary. Except for the bounce or non-deliverable messages, bitching
about the inability to send emails to non-existant accounts across the
internet (XXXXXXX spammers) and for those, I funnel
those into a single folder (using procmail) which I then delete
on a daily basis (why don't I just delete them as I get them? There is the
rare occasion when I need to check those messages and because of that, it's
easier to just delete them manually than recofiguring the email client when
I need to actually search for those messages).
So it was surprising to find such messages in my default inbox. More surprising was the sheer number: over a thousand.
And not only in my default inbox, but in just about every incoming email folder. Hundreds. Thousands. Jamming up my email.
Checking the procmail log file, I found thousands
of instances of:
procmail: Error while writing to "in-mailerdaemon" procmail: Truncated file to former size procmail: Error while writing to "in-mailerdaemon" procmail: Truncated file to former size
Hmmm …
The folder itself was 51,200,000 bytes in size. Inside were nearly 6,000 messages, all the bitching about the same thing:
----- The following addresses had permanent fatal errors ----- <root@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX> (reason: 554 5.4.6 Too many hops) <admin@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX> (reason: 554 5.4.6 Too many hops) ----- Transcript of session follows ----- 554 5.4.6 Too many hops 26 (25 max): from <root@XXXXXXXXXXXXXXXXXX> via localhost, to <root@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
which was trying to inform me that it couldn't deliver the following email message:
- From
- root <root@XXXXXXXXXXXXXXXXXX>
- To
- admin@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
- Subject
- CRITICAL:Current disk space consumption has reached a critical limit.
- Date
- Sat, 18 Nov 2006 04:23:27 -0500
- Cc
- root@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
- X-Yoursite-Mailscanner
- Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details
- X-Yoursite-Mailscanner-Information
- Please contact the ISP for more information
- X-Mailscanner-From
- root@XXXXXXXXXXXXXXXXXX
What's even funnier is that this is a customer of one of our resellers. Even funnier is that the site in question has been suspended for non-payment.
Wheee!
So I delete the overflowing inbox, and the incoming email starts flowing properly again, and I start investigating the problem further.
A few hours go by, and I start finding bounce messages in the wrong inboxes. Same error. And the folder is again 51,200,000 bytes in size.
Gee, guess I found some internal procmail limit or bug or
something.
In the meanwhile, I find out that not only does the site belong to one of our resellers, and has been suspended, it has also moved servers!
Wonderful!
And the mail is originating from the original server, bouncing around the new server and after bouncing around too many times, bounces towards me.
Now, have I mentioned I hate control panels?
Yup, they show up here, because the makers of Insipid have
modified sendmail to handle their method
of virtual servers and the command I'm used to using to check mail queues,
mailq, is broken. So I have to track down where on God's Green
Server they stuck the outgoing email queue and clean it up on two
servers.
And yet I'm still getting bounce messages from the server, dating back to Saturday!
It was then I realized that our Office Email goes through yet another server, a spam firewall appliance. And it probably has a huge backlog of messages headed my way. And unfortunately, there's no way to flush the outgoing queue, because we have no access, because it's an appliance.
Sigh.
So for now, I have the email directed towards the bit bucket. I just hope that for the next few days, I don't miss a really important message.
