Tuesday, August 06, 2019
There are even bots crawling gopherspace
My webserver isn't the only program beset by bots—my gopher server is also being crawled.
I identified one bot repeatedly trying to request the selector
(the gopher equivalent of a web page) Phlog when it should be trying to request Phlog:
(note the ending “:”).
On the web server,
I could inform the client of the proper link with a “permanent redirect” and hope it gets the hint,
but gopher lacks such a facility.
All this bot was getting back was the rather lack luster gopher error,
which for an automated process,
is pretty darned hard to distinguish from actual content,
due to the simplicity of the protocol.
Oh a lark, I decided to see if there was a gopher server on the IP address of the bot, and lo', there was. I was able to send an email to the organization responsible, and they fixed the error.
That still left a few bots that thought I was running a web server on port 70.
Yes,
I was getting requests for “GET / HTTP/1.1” over and over again,
and these particular bots weren't getting the clue they weren't talking to a web server by the lack of proper web server response.
I decided to handle these by replying as a tea pot because why not?
And to further support the joke,
my gopher server will not only respond to the web method GET but also BREW
(and to think I wanted to write a gopher server,
not a web server … sigh).
Hopefully that will placate them and they'll go away
(although on second thought,
I think I should have done a permament redirect to gopher://gopher.conman.org/ to see how well the web bots would handle that!).
An MJ12Bot update
When last we left the MJ12Bot saga, it was pretty apparent it wasn't a well written bot, but true to their word, they haven't crawled my server since.
“The Knowledge AI” bot however …
it is trying to repeatedly fetch /%22https:/mj12bot.com/%22 from my web server.
What is it with these horribly written web bots?
