Wednesday, September 03, 2008
Owning more than you expect
A few years ago I set things up at The Company (formerly known as The Company) so that emails to
root get sent to me, which means, I get all the emails the
servers automatically generate as they're running. Prior to that, all email
to root was slowly being accumulated on the various servers
without being checked.
Needless to say, I get an inordinate amount of mail at work, I can scan
it quickly (which is the reason why I
switched to using mutt) and generally keep on top of
things.
I've learned, for instance, that when I get an email like:
- From
- support@XXXXXXXXXXX
- To
- support@XXXXXXXXXXX
- Subject
- URGENT: MailQueueCleaner failure report
- Date
- Tue, 2 Sep 2008 19:37:45 -0400
This is a report generated by the hourly run of MailQueueCleaner. Some sites on this server have problems which prevent the processing of their email queues. UNTIL YOU RESOLVE THE PROBLEMS, THESE SITES WILL NOT BE ABLE TO RECEIVE EMAIL. A summary of the problems is included below.
XXXXXXXXXXXXXXXXXXXXXXXXX: over disk quota, usage: 648180K, quota: 614400K
I know that one of our customers has a catch-all email address that is never being checked and I can pretty much nuke emails to said address (actually, I don't—I empty the mailbox of all the emails, which are then backed up in case the customer actually does need them, but in the several years we've been doing this, the customer has never noticed nor asked to scan the email backups). This used to happen quite often, but since installing X-Grey this now happens rarely.
But for the past month now, I've been receiving a large amount of these emails all from the same account (in fact, the example above is one such email). When I first received it, I did what I always do in such situations, and yet, that didn't resolve the issue.
We've been poking into this, trying to find out where this customer's quota is going; it certainly isn't due to spam—their email is practically empty. Disk usage over their account shows nothing too outrageous and well within the limits.
Very puzzling indeed.
Until P did a very deep scan of the entire filesystem looking for files that might be owned by this customer, and found a metric-buttload of files in a rather unexpected location.
Earlier this year I wrote a Perl-wrapper to help track down some exploited sites. This Perl-wrapper records the script being run (it maintains a single copy), any files it may be using, and information about the script (which user is running it, working directory, environment, etc). And this happens for every Perl script on the system!
But when I wrote it, little did I realize that the files it created would be owned by the group the user that executed the script belonged to. And modern Linux usage tends to give each user their own private group.
Oh.
So that's what's blowing their quota.
Heh.
(Simple fix actually—change the user and group ownership of the files the Perl-wrapper creates to be something other than the user. In this case, that was two lines to fix the problem).
