Wednesday, February 10, 2010
Notes on logging
This is interesting: Facebook wrote their own logging system instead of using syslog. Their system only has two pieces of informtion—a catagory and the message. No facilities, no priorities or levels. I think in Facebook's case, they log everything so there's no need for individual priorities or levels (the argument here is: you're going to log everything eventually anyway, so simplify the process).
Another note: when your configuration file is too complex (or in other words—an ad-hoc declarative language) perhaps it's time to give up and just use a scripting language for configuration (I skipped straight to using a scripting language for configuration/logic).
More notes on logging
I mentioned yesterday about logging
all mail related logs to a central server. While we don't have a
complicated email setup (unlike, say, Negiyo), we still have several email
severs and we get enough tickets about slow or lost email that it's a pain
having to slog through one or two servers piecing everything together. What
I would like is, given a Message-ID (which is (supposed to be)
a globally unique identifier for an email) or an email address, to make a
query in one location and get something like:
message-id = <YzNCeWFXNW5RSE53Y21sdVoyUmxkeTVqYjIwPQo=@mx3oc.com> from = gandalf@example.net to = sean@example.com [rhohan-isp.example.org] [gondor.example.net] Feb 10 22:46:56 [gondor.example.net] [spamfirewall.example.com] Feb 10 22:46:57 [spamfirewall.com] [compmailserv.example.com] Feb 10 22:47:02 [compmailserv.example.com] [workstation.example.com] Feb 10 22:47:06 [workstation.example.com] mbox of sean Feb 10 22:47:06
As an example, you see the Message-ID, who sent the email,
who received it, and the five other lines can be read as: machine X sent
email to machine Y at such-n-such a time,” with the last one showing local
delivery of the email to a mailbox.
Anyway, that's what I would like to build. And I can almost do it. Sendmail (which at
The Company we use on our legacy systems), Postfix (which we use for new servers)
and Exim (which we use
on one server because it has a feature that's needed by a program
that runs on that one server) all log a bunch of messages as email works
through their respective systems. Each one uses an internal unique ID,
but they at least log the Message-ID at some point, so
I can map the respective MTAs internal IDs to a globally unique ID.
The odd-man out though, is our spam firewall, which is used by a
significant portion of our customers. But, given that our spam firewall is
OpenSource™ I suppose I can modify the source code to emit a
Message-ID, but the problem there is if (or when) we
upgrade—I would have to patch the code again (or, convince the Powers That
Be to accept the patch).
I would also like to convert as many software packages to log via
syslog, and while most, like PostgreSQL and even Apache, can be
configured as such, there are a few holdouts (I'm looking at you,
MySQL) that can't.
