The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Tuesday, August 06, 2019

There are even bots crawling gopherspace

My webserver isn't the only program beset by bots—my gopher server is also being crawled. I identified one bot repeatedly trying to request the selector (the gopher equivalent of a web page) Phlog when it should be trying to request Phlog: (note the ending “:”). On the web server, I could inform the client of the proper link with a “permanent redirect” and hope it gets the hint, but gopher lacks such a facility. All this bot was getting back was the rather lack luster gopher error, which for an automated process, is pretty darned hard to distinguish from actual content, due to the simplicity of the protocol.

Oh a lark, I decided to see if there was a gopher server on the IP address of the bot, and lo', there was. I was able to send an email to the organization responsible, and they fixed the error.

That still left a few bots that thought I was running a web server on port 70. Yes, I was getting requests for “GET / HTTP/1.1” over and over again, and these particular bots weren't getting the clue they weren't talking to a web server by the lack of proper web server response. I decided to handle these by replying as a tea pot because why not? And to further support the joke, my gopher server will not only respond to the web method GET but also BREW (and to think I wanted to write a gopher server, not a web server … sigh). Hopefully that will placate them and they'll go away (although on second thought, I think I should have done a permament redirect to gopher:// to see how well the web bots would handle that!).

An MJ12Bot update

When last we left the MJ12Bot saga, it was pretty apparent it wasn't a well written bot, but true to their word, they haven't crawled my server since.

“The Knowledge AI” bot however … it is trying to repeatedly fetch /%22https:/ from my web server.

What is it with these horribly written web bots?

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.