The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, Debtember 22, 2021

Schadenfreude at some badly written bots

Back in October, I removed a whole section of my Gemini site because I got fed up with badly written bots. In the best practices guide there is a section about redirection limits (something that should be in the specification but for some reason isn't). In the section of my site I removed, I had such a test—a link that would always redirect to a link that itself would redirect, ad nauseum. So I would see a new client being tested and perhaps a dozen or two attempts to resolve the link and then it would stop.

But then there were the clients written for automated crawling of Gemini space that never bothered with redirection limits. It got so bad that there was one client stuck for a month, endlessly making requests for a resource that was forever out of reach. I had even listed the redirection test as a place for bots NOT to go in the robots.txt file for the site (also a companion specification).

So because it can be needlessly complex to locate a bot contact address, I was like, XXXX it. I removed the redirection test (along with the rest of the section, which included a general Gemini client test because, well, XXXX it). Why waste my bandwidth for people who don't care?

I only bring this up now because I'm noticing two bots who are attempting to crawl a now non-existant redirection test, probably with a backlog of thousands of links because hey, who tests their bots anyway?

Obligatory Picture

An abstract representation of where you're coming from]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

Obligatory AI Disclaimer

No AI was used in the making of this site, unless otherwise noted.

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.