Thursday, August 14, 2003
The Archive Effect
I just implemented a small change to mod_blog, the code that
runs this site. I added a new template command,
%{robots.index}% that runs the following fragment of code:
if (m_navunit == PART) fprintf(fpout,"index"); else fprintf(fpout,"noindex");
To use it:
<META NAME="robots" CONTENT="%{robots.index}%,follow">
The effect is to generate:
<META NAME="robots" CONTENT="noindex,follow">
or
<META NAME="robots" CONTENT="index,follow">
Depending on if the page contains multiple entries or a single entry.
So far, nothing earth shattering and to tell the truth, pretty trivial as features go. Not to mention pretty specific. This particular tag is used to control the behavior of search engine robots (at least, those that understand this convention) and it may seem strange that I am instructing these web robots to skip indexing my site unless they pull up a single entry.
But it's not that strange when you think about it. And the feature, however trivial it is to implement (and it's been a feature I've been wanting to add for some time now) is a form of search engine optimization. And is something that other blogging software should do, but doesn't.
First off, the chance of someone coming across my pages with a disturbing search request is pretty high, given that up to a full year's worth of text can be indexed (actually, you can get the every entry I've written here in one shot, but I'll leave that up as an exercise for the reader) so that I get a search like:
the chipper that is configured like a triangle with two hitting areas is illegal
with the resultant page being all the entries I've written for 2001; I'm guessing that most, if not all, of those words do appear within the entries of 2001, but not in a single entry.
The result isn't just to solely remove the posibility of distrubing search requests from my site (“aggresiveness of college students in manila”—I'm not a college student, nor am I in Manila, but hey, if they're female, I wouldn't mind them being aggressive) but to actually help the search engines generate good results for people (I'm sure the person looking for “door king 1812 dsl technical information” didn't appreciate being fed every entry for 2002).
Then again, I will miss those that come looking for “mitubishi polyester film corp.”
