The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Tuesday, March 24, 2009

Rube Goldberg would have love modern software development

“In a system of a million parts, if each part malfunctions only one time out of a million, a breakdown is certain.”

Stanislaw Lem

“Project: Leaflet” is a web based application written in PHP, so there's a dependency on a webserver (in our case, Apache) and PHP itself. There's also a database involved, so throw in MySQL, and, oh heck, Linux to run the whole mess, and you have a pretty standard LAMP stack going on.

Although in our case, we use PostgreSQL (and yes, I have two active versions of “Project: Leaflet”) so that means we have a pretty standard LAPP stack going on.

Now, “Project: Leaflet” is designed to help manage mass emails, like a newsletter, or a product announcement, or perhaps to notify customers of an impending company name change, so it requires an SMTP server of some sort. At first we used whatever SMTP server came with the distribution of Linux we used (at first sendmail but now we also use postfix).

Then came the automatic handling of bounced emails, which meant a bit more involvement with the SMTP server (sending emails to a particular address to a program). It also meant parsing the incoming message and given that there really isn't a standard for bounce messages (or rather, quite a few, take your pick) the stock configuration for the SMTP server wasn't going to cut it anymore.

First off, we needed to enable address extensions, and at least under postfix, that's just adding one line to the configuration, but it's still a change from the default install (such a configuration may be enabled by default under sendmail, but we don't run “Project: Leaflet” with sendmail, so it's a moot point). The point of the address extension is to encode the outgoing email address in our return address so we don't actually have to parse the body of the bounce.

It also helps with handling confirmation replies.

But there's still the issue of parsing the headers from the incoming email, and well, I know procmail, and I know that's what it's there for, to help in dealing with incoming emails. I mean, it's there right? (Well, actually, not by default any more. And that means yet more changes to the SMTP configuration to enable procmail processing). And yes, the syntax is horrible, but it's the very model of clarity compared to some other syntaxes you come across on a Unix system (like, oh, anyone?).

So now the dependencies for “Project: Leaflet” are: Linux, Apache, PostgreSQL, PHP, sendmail or postfix (with some custom configurations) and procmail.

Oh, and then there's the change from a few weeks ago, where I made it such that a single copy of the code can serve multiple instances (clients) on a single server. I did that not to save disk space (heaven's no! The uncompressed source code doesn't even come close to a megabyte of disk space) but administrative overhead—I can update the code in one location per server instead of, say, 25 individual copies.

But that necessitated some changes in layout and assumed locations of files and well … let's just say that while a company could take our code base (and at one point the topic of releasing the code as Open Source™ did come up), it's less and less likely that an average website owner would be able to install this program, given the dependencies and somewhat custom configurations made thus far.

Now, this version (the “Install Once And Use Everywhere On A Server” version) mandated yet another “feature”—support of a per-client IP address.

We have the IP addresses. No problem there.

And Linux supports multiple IP addresses. No problem there.

And Apache supports multiple IP address. No problem there.

sendmail/postfix support multiple IP addresses. No probl—

Uh … hold on a second …

Er, mu.

Incoming email isn't an issue—sendmail/postfix can be set up to listen on all interfaces. It's outgoing email that's an issue.

See, it's all too easy for certain ISPs to blacklist IP addresses that send an excessive amount of spam (and that happens to us all the time), so if that happens, we want the customer's assigned IP address to get banned, and not the IP address of the server (otherwise, all our customers on said server have “an issue”).

But, when sendmail/postfix send email, they use the default server IP address, which isn't what we want.

So, mu.

After some searching, we found that exim will do what we want (great! Yet another SMTP server we need to learn and support), but not out of the box (nor will it support address extensions, virtual hosts or procmail, so there're quite a bit of configuration changes going on).

So, this little PHP app I wrote requires Linux, Apache, PHP (of course), PostgreSQL, exim and procmail. Talk about your debugging nightmare.

Today, I spent several hours tracking down an issue where emails were being sent out via the server's primary IP address and not the client's IP address. That meant I had to track the issue through half the chain there, PHP, exim and procmail. I ended up chasing (now that I look at it after the fact) a potential red herring. The problem was compounded by the fact that it worked for the test account.

I fixed the issue by replacing the call to the PHP mail() function (which calls the external program sendmail—which isn't really sendmail but a simple replacement provided by exim, which injects the email directly into the outgoing queue) with a bunch of PHP code that talks to the local email server via SMTP.

That worked. Only later, I realized that the client was using the incorrect email address for sending emails, which was probably why the primary IP address was showing up when it shouldn't (and if that makes no sense to you, Welcome to My World!).

It certainly feels as if the modern web development world is held together with chewing gum and bailing wire (and we're all out of chewing gum) and nobody really understands what's going on underneath the covers (and frankly, are as scared of it as they are of seeing how sausage is made).

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2023 by Sean Conner. All Rights Reserved.