The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, September 03, 2008

Owning more than you expect

A few years ago I set things up at The Company (formerly known as The Company) so that emails to root get sent to me, which means, I get all the emails the servers automatically generate as they're running. Prior to that, all email to root was slowly being accumulated on the various servers without being checked.

Needless to say, I get an inordinate amount of mail at work, I can scan it quickly (which is the reason why I switched to using mutt) and generally keep on top of things.

I've learned, for instance, that when I get an email like:

From
support@XXXXXXXXXXX
To
support@XXXXXXXXXXX
Subject
URGENT: MailQueueCleaner failure report
Date
Tue, 2 Sep 2008 19:37:45 -0400

This is a report generated by the hourly run of MailQueueCleaner. Some sites on this server have problems which prevent the processing of their email queues. UNTIL YOU RESOLVE THE PROBLEMS, THESE SITES WILL NOT BE ABLE TO RECEIVE EMAIL. A summary of the problems is included below.

XXXXXXXXXXXXXXXXXXXXXXXXX: over disk quota, usage: 648180K, quota: 614400K

I know that one of our customers has a catch-all email address that is never being checked and I can pretty much nuke emails to said address (actually, I don't—I empty the mailbox of all the emails, which are then backed up in case the customer actually does need them, but in the several years we've been doing this, the customer has never noticed nor asked to scan the email backups). This used to happen quite often, but since installing X-Grey this now happens rarely.

But for the past month now, I've been receiving a large amount of these emails all from the same account (in fact, the example above is one such email). When I first received it, I did what I always do in such situations, and yet, that didn't resolve the issue.

We've been poking into this, trying to find out where this customer's quota is going; it certainly isn't due to spam—their email is practically empty. Disk usage over their account shows nothing too outrageous and well within the limits.

Very puzzling indeed.

Until P did a very deep scan of the entire filesystem looking for files that might be owned by this customer, and found a metric-buttload of files in a rather unexpected location.

Earlier this year I wrote a Perl-wrapper to help track down some exploited sites. This Perl-wrapper records the script being run (it maintains a single copy), any files it may be using, and information about the script (which user is running it, working directory, environment, etc). And this happens for every Perl script on the system!

But when I wrote it, little did I realize that the files it created would be owned by the group the user that executed the script belonged to. And modern Linux usage tends to give each user their own private group.

Oh.

So that's what's blowing their quota.

Heh.

(Simple fix actually—change the user and group ownership of the files the Perl-wrapper creates to be something other than the user. In this case, that was two lines to fix the problem).

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2019 by Sean Conner. All Rights Reserved.