The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Tuesday, May 01, 2007

Which version was I using again?

Sorry if you've signed up for email notification and gotten some wierd notifications over the past twelve hours. That's because there was a show-stopping bug in the code that only now was triggered.

You see, I rewrote the email notification code some time ago, but hadn't fully tested it yet. Then, a few days ago I decided to add in custom error pages and kept having issues. I knew I had done it before, then I realized I did, over there, but the custom error pages weren't working over here. I thought maybe it was a different version of the code.

So I checked out a fresh copy over at the other place to see if I broke whatever it was that made the custom pages work (it was using an older code base there), and it still worked. But since I'm not using email notification over there, the new email notification code was still untested.

So I checked out a fresh copy here. The custom error pages were still broken.

Turns out it was the installation of Apache on the server here (I used the distro-installed version for a change)—seems that it has a directory alias for /error/, which just so happens is what I named the directory. Had I named it /errors/ (plural) things would have worked the first time and all of this wouldn't have happened (or rather, would have happened at a later time).

Sigh.

But I did.

But notice that I hadn't tested the email notification code yet.

Until yesterday and today.

And boy, did things blow up.

The program kept crashing. Normally, that isn't much of an issue (well, okay, it is an issue) because normally, one is in front of the program to watch it crash, but in this case, I wasn't aware of the crash, because I used the email interface.

Now, mod_blog takes the entry (from a web page, a file, or through email), adds it to the archive, generates the new page, and then, if enabled, sends the email notifications.

It was crashing (unbeknowst to me) during the email notifications.

Postfix has been instructed to send emails to a particular address to mod_blog, which it does. But Postfix also notices that the program crashed. So it requeues the email.

And when the program crashes, the update lock it has is released (technically, it's a file lock that the kernel maintains, but hey, the program goes away, so does the lock).

Next entry comes along, process repeats, and Postfix realizes it has some email queued up, and tries to send that at the same time. Now we have two processes trying to update the blog. Normally, this isn't an issue because of the update lock. But the lock is released in an uncontrolled state and …

Things blow up and get ugly.

But what I'm seeing is a blog that failed a validation check, I fix it, only to have it fail the validation check again, which I fix, only to fail the validation yet again, only now I notice that the same entry I thought I fixed twice keeps coming back to haunt me and even worse, the entire front page is a mess, and is remaining a mess no matter what I do.

I tracked the problem down, but I may have sent some spurious notifications in doing so. Once I realized it was the new email notification code (and man, how did I ever check that code in? Leaks memory like you wouldn't believe! Sheesh) I reverted to the old version, cleared out the mail queue and hopefully, got this situation under control.

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.