Wednesday, October 28, 2026
Can the Robot Exclusion Protocol apply to Gopher URLs?
A month ago,
slugmax wrote a post about
LLMs invading gopher via an MCP server
and now that MCP server supports the /robots.txt standard.
He gives a sample file that should work for Gopher:
User-agent: gopher-mcp Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: Claude-User Disallow: / User-agent: Google-Extended Disallow: /
There are problems with this advice.
First off,
Gopher doesn't support additional information per a request,
unlike HTTP.
There is no way a Gopher client can identify itself to a Gopher server.
Well,
there is a hack one could use: include a search query when retrieving the /robots.txt file,
but the current Gopher standard says little about how a server responds to a search query on a non-search-query selector.
I mean,
it's probably okay as most servers will (hopefully?) ignore the search query on a non-search query selector.
And the above list will (hoefully?) work for the given LLMs,
but what about a new LLM not in the list above?
How do you learn about those?
Gemini, which also doesn't have a way for clients to identify themselves, found a clever workaround: specify a virtual user agent based on the type of work:
archiverfor agents archiving Gemini;indexerfor agents indexing Gemini;researcherfor agents studying statistical properties of Gemini.webproxyfor agents bridging Gemini to HTTP;
I suppose one could add llm (or ai) for the current crop of LLM agents to this list.
But this doesn't solve the larger issue with using /robots.txt with Gopher: Gopher selectors are opaque strings:
Servers return either directory lists or documents. Each item in a directory is identified by a type …, user-visible name …, an opaque selector string …, a host name …, and an IP port number …
RFC 1436: The Internet Gopher Protocol (emphasis added)
Unlike HTTP and Gemini URLs where the “selector” has an actual structure,
Gopher selectors aren't defined as such.
They are to be treated as a block of characters that,
went sent as is to a Gopher server,
return data.
In fact,
Gopher URLs aren't even compatible with HTTP or Gemini URLs and have their own RFC.
Gopher “links”
(because they aren't “links” in the way that a HTTP or Gemini define them)
contain the type of data to be served to inform the client what to expect.
Because of this,
the Gopher URL has to include this “type” information.
Furthermore,
a Gopher selector doesn't have to start with a leading '/',
or even have a '/' in it!
So the Gopher URL for my blog is gopher://gopher.conman.org/1Phlog:.
While there is a '/' separating the domain from the selector,
that's it.
The '/' there is required per the Gopher URL RFC,
but it's not part of the selector like it is for the “path” for HTTP or Gemini.
The “1” is the type
(in this case,
it designates that the document is a “Gopher map” which is links to further documents),
and is not part of the selector either.
It just exists to inform the client what to expect.
The actual selector sent to the Gopher server is “Phlog:”.
And that's the problem.
That example /robots.txt above just won't work on my Gopher server.
An empty Disallow: line explicitly indicates “nothing is disallowed”.
A Disallow: line of “/” for HTTP and Gemini explicitly indicates “everything is disallowed”
(given how /robots.txt defines matches).
I don't even know how one would disallow selectors for Gopher in /robots.txt.
The leading “/” and type character in a Gopher URL aren't part of the selector,
and in the /robots.txt standard,
all possible matches start with a leading “/” because all “selectors” for HTTP and Gemini
do start with a leading “/”.
If I'm find with disallowing all crawling by an LLM,
then doing a “Disallow: ” would be fine,
but if I wanted to prevent an LLM from going into certain parts of my Gopher site,
such as gopher://gopher.conman.org/1BlackHole:,
there is no way I could actually specify it.
I can't specify “Disallow: /Blackhole:” because that selector doesn't start with a “/”.
And I don't think “Disallow: /1Blackhole” would work because the “1” isn't part of the selector either.
The agent can't just ignore the leading “/”,
because if a selector does start with a “/”,
then the entry in the /robots.txt file would have to be “Disallow: //foo”.
And I suspect I'm overthinking this.
I just checked the /robots.txt on my Gopher server and forgot I had this:
User-agent: * Disallow: Blackhole:
Even though there's a “link” to that selector on the main page, no one crawls through it. So perhaps it's working? Maybe?
Yeah,
while I think I'm overthinking on this,
I also don't think /robots.txt really can apply to Gopher in a general sense
(or rather—my Gopher server in particular since my selectors generally don't start with “/”).
![Oh Chrismtas Tree! My Christmas Tree! Rise up and hear the bells! [Self-portrait with a Christmas Tree] Oh Chrismtas Tree! My Christmas Tree! Rise up and hear the bells!](https://www.conman.org/people/spc/about/2025/1203.t.jpg)