Saturday, January 12, 2019
It's no longer possible to write a web browser from scratch, but it is possible to write a gopher browser from scratch
As I mentioned two months ago, I've been browsing gopherspace. At the time, I was using an extension to Firefox to browse gopherspace, but a recent upgrade to Firefox left it non-working. I could use Lynx but it includes unnecessary pauses that make it feel slower than it really should be. I also don't care for how it lays out the page.
So I've been writing my own CLI gopher client.
And it's not like the protocol is all that difficult to handle and everything is plain text.
How hard could it be to download a bunch of text files and display them?
The protocol? Trivial.
Displaying the pages? Er … not so trivial.
The first major problem—dealing with UTF-8.
Problem—the terminal window can only display so many characters per line
(default of 80 usually).
There are two ways of dealing with those—one is to wrap the text to the following lines(s),
and the other is to “pan-and-scan”—let the text disappear off the screen and pan left-and-right to show the longer lines.
Each method requires chopping text to fit though.
With ASCII, this is trivial—if the width of the temrinal is N columns wide,
just chop the line at every N-bytes.
This works because each character in ASCII is one byte in size.
But characters in UTF-8 take a variable number of bytes,
so chopping at arbitrary byte boundaries is less than optimum.
The solution may look simple:
-- ************************************************************************ -- usage: writeslice(left,right,s) -- descr: Write a portion of a UTF-8 encoded string -- input: left (integer) starting column (left edge) of display screen -- right (integer) ending column (right edge) of display screen -- s (string) string to display -- ************************************************************************ local function writeslice(left,right,s) local l = utf8.offset(s,left) or #s + 1 local r = utf8.offset(s,right + 1) or #s + 1 tty.write(s:sub(l,r - 1)) end
but simple is never easy to achieve. It took a rather surprising amount of time to come up with that solution.
The other major problem was dealing with the gopher index files. Yes, they are easy to parse (once you wrap your head around some of the crap that presents itself as a “gopher index” file) but displaying it was an even harder problem.
Upon loading a gopher index,
I wanted the first link to be highlighted,
and use the Up and Down keys to select the link and then the enter key to select a page to download and view.
Okay,
but not all lines in a gopher index are actual links.
In fact,
there are gopher index files that have no actual links
(that surprised me!).
And how do I deal with lines longer than can be displayed?
Wrap the text?
Let the text run off the screen?
At first, I wanted to wrap long lines, but then trying to manage highlighting a link that spans several lines when it might not all be visible on the screen (the following lines might be off the bottom, for instance) just proved too troublesome to deal with. I finally just decided to let long lines of text run off the end of the screen just to make it easier to highlight the “current selection.” Also, most gopher index pages I've come across in the wild generally contain short lines, so it's not that much of a real issue (and I can “pan-and-scan” such a page anyway).
For non-text related files,
I farm that out to other programs via the mailcap facility found on Unix systems.
That was an interesting challenge I will probably address at some point.
There are still a few issues I need to address, but what I do have works. And even though it's written in Lua it's fast. More important, I have features that make sense for me and I don't have to slog through some other codebase trying to add an esoteric feature.
And frankly, I find it fun.
The technical differences between HTTP and gopher
… The point is to attempt as full a sketch as possible of the actual differences and similarities between the HTTP and GOPHER protocols.
…
From what I gather, these are the similaries:
- Both gopher and http start with a TCP connection on an IANA registerd port number.
- Both servers wait for text (the request) terminating in a CRLF
- Both servers expect the request (if there is one) to be formatted in a particular way.
- Both servers return plain text in response, and close the TCP connection.
And these are the differences that I understand:
- Gopher will accept and respond to a blank request, with a default set of information, http will not.
- Gophper [sic] sends a single "." on a line by itself to tell the client it is done, http does nothing similar prior to closing the connection.
- Http has things like frames, multiplexing, compression, and security; gopher does not.
- Http has rich, well-developed semantics, gopher has basic, minimalist semantics
- Http requests are more resource intensive than gopher requests.
- Http is highly commercialized, gopher is barely commercialized.
- Http is heavily used and highly targeted by malicious users, gopher is neither.
- Http is largely public, gopher is largely private (de facto privacy through obscurity.)
- Http is used by everyone, their children, their pets, their appliances, their phones, and their wristwatches; gopher is used primarily by technical folk and other patient people.
- Http all but guarantees a loss of privacy; gopher doesn't
Yeah, I know, it's not much, but that's all that is coming to mind presently. What are your thoughts?
Tech
nology/Gopher (I'm quoting for the benefit of those that
cannot view gopher based sites).
I don't want to say that tfurrows is wrong, but
there is quite a bit that needs some clarification, and as someone who has
worked with HTTP for over
twenty years, and has recently dived back into gopher (I used it for several
years in the early 90s—in fact, I recall Time Magazine having a gopher server back then) I
think I can answer this.
First, the protocol. The gopher protcol is simple—you make a TCP connection to the given port (defaults to 70). Upon connection, the client then sends the request which can be one of three formats:
CRLF
The simplest request—just a carriage return and line feed character. This will return the main page for the gopher server.
selector-to-viewCRLF
This will return the requested data from the gopher server. The specification calls this a “selector.” And yes, it can contain any non-control character, including space. It's terminated by a carriage return and line feed characters.
selector-for-searchHTsearch terms to useCRLF
The last one—this sends a search query to a gopher server. It's the “selector” that initiates a search, followed by a horizontal tab character, then the text making up the query, followed by a carriage return and line feed.
In all three cases, the gopher server will immedately start serving up the data. Text files and gopher indexes will usually end with a period on its own line; other file transfers will end with the server closing the connection.
That's pretty much the gopher protocol.
The HTTP protocol that works the closest to gopher is the so called HTTP/0.9 version, and it was pretty much the the same. So the same three requests above as HTTP requests.
GET /CRLF
The minimum request for HTTP. As you can see, it's only an extra four characters,
but the initial text, GET in this case, was useful later when
the types of requests increased (but I'm getting ahead of myself here). This
will return the main page for the HTTP server.
GET /resource_to_viewCRLF
The usual request, but instead of a “selector” you request a “resource”
(different name, same concept) but it cannot contain bare spaces—they have to
be encoded as %20 (and a bare “%” sign is encoded as
%25). Like gopher, the contents are immediately sent, but there
is no special “end-of-file” marker—the server will just close the
connection.
GET /resource_for_seach?search%20terms%20to%20useCRLF
And a search query, where you can see the spaces being replaced with
%20. Also note that the search query is separated by the
“resource” with a “?”.
So not much difference between gopher and HTTP/0.9. In fact, during the early to mid-90s, you could get gopher servers that responded to HTTP/0.9 style requests as the difference between the two was easy to distinguish.
The next version of HTTP, HTTP/1.0, expanded the protocol. Now, the client was expected to send a bit more infomration in the form of headers after the request line. And in order to help distinguish between HTTP/0.9 and HTTP/1.0, the request line was slightly expanded. So now the request would look like:
GET /resource_to_view HTTP/1.0CRLF User-Agent: Foobar/1.0 (could be a web browser, could be a web crawler)CRLF Accept: text/*, image/*CRLF Accept-Language: en-US;q=1.0, en;q=0.7; de;q=0.2, se;q=0.1CRLF Referer: http://www.example.net/search?for%20blahCRLF CRLF
(Yes, “Referer” is the proper name of that header, and yes, it's mispelled)
I won't go too much into the protocol here, but note that the client can
now send a bunch more information about the request. The Accept
header now allows for so-called “content negotiation” where the client
informs the server about what type of data it can deal with; the Accept
Language header tells the server the preferred languages (the example
above says I can deal with German, but only if English isn't available, but
if English is availble, American is preferred). There are other
headers; check the specification for
details).
The server now returns more information as well:
HTTP/1.0 200 OkayCRLF Date: Sun, 12 Jan 2019 13:39:07 GMTCRLF Server: Barfoo/1.0 (on some operating system, on some computer, somewhere)CRLF Last-Modified: Tue, 05 Sep 2017 02:59:41 GMTCRLF Content-Type: text/html; charset=UTF-8CRLF Content-Length: 3351CRLF CRLF content for another 3,351 bytes
The first line is the status, and it informs the client if the “resource”
exists (in this case, a 200 indicates that it does), or if it
can't be found (the dreaded 404) or if it has explicitely been
remove (410) or it's been censored due to laws
(451), or even moved elsewhere.
Also added were a few more commands in addition to GET, like
POST (which is used to send data from the client to the server)
and HEAD (which is like GET but doesn't return any
content—this can be used to see if a resource has changed).
HTTP/1.1 is just more of the same, only now you can make multiple requests per connection, a few more commands were added, and the ability to request portions of a file (say, to resume a download that was cut off for some reason).
HTTP/2.0 changes the protocol from text-based to binary (and attempts to do TCP- over-TCP but that's a rant for another time) but again, it's not much different, conceptually, than HTTP/1.1.
Security, as in https: type of security, isn't inherently
part of HTTP. TLS is basically inserted between the
TCP and HTTP layers. So the same could be
done for gopher—just insert TLS
between TCP and gopher and
there you go—gophers:. Of course, that now means dealing with
CAs and certificates and
revocation lists and all that crap, but it's largely orthogonal to the
protocols themselves.
HTTP/1.0 allows compression but that falls out of the content negotiation. The bit about frames and multiplexing is more an HTTP/2.0 issue which is a lot of crap that the server has to handle instead of the operating system (must not rant …).
Are HTTP requests more resource intensive? They can be, but they don't have to be. But that leads right into the commericalization of HTTP. Or rather, the web. HTTP is the conduit. And conduits can carry both water and waste. HTTP became commercialized because it became popular. Why did HTTP become popular and gopher whithered? Personally, I think it has to do with HTML. Once you could inline images inside an HTML document, it was all over for gopher. The ability to include cat pictures killed gopher.
But in an alternative universe, where HTML had no image support, I think you would have seen
gopher expand much like HTTP
has. Work was started in 1993 to to expand
the gopher protocol (alternative link) where the protocol gets a
bit more complex and HTTP-
like. As mentioned, a secure gophers: is “easy” to add in that it doesn't change the core
protocol (update—it's not as easy
as I thought). And as such, I could see it getting more commercialized.
Advertising can be inserted
TYPEWRITERS
For SALE, HIRE, or EXCHANGE,
at HALF the USUAL PRICES.
MS.
Typewritten from
10d. per 1,000 words. 100 Circulars for 4s
TAYLOR'S,
74, Chancery Lane, London.
(Est. 1884.)
Telegrams: "Glossator," London.
Telephone No. 690,
Holborn.
even in a text file. Yes, it might look a bit strange, but it can be done. The only reason it hasn't is that gopher lost out to HTTP.
So those are the differences between HTTP and gopher. HTTP is more flexible but more complex to implement. Had history played out differently, perhaps gopher would have become more flexible and complex.
Who knows?
