The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Saturday, January 12, 2019

It's no longer possible to write a web browser from scratch, but it is possible to write a gopher browser from scratch

As I mentioned two months ago, I've been browsing gopherspace. At the time, I was using an extension to Firefox to browse gopherspace, but a recent upgrade to Firefox left it non-working. I could use Lynx but it includes unnecessary pauses that make it feel slower than it really should be. I also don't care for how it lays out the page.

So I've been writing my own CLI gopher client.

And it's not like the protocol is all that difficult to handle and everything is plain text.

How hard could it be to download a bunch of text files and display them?

The protocol? Trivial.

Displaying the pages? Er … not so trivial.

The first major problem—dealing with UTF-8. Problem—the terminal window can only display so many characters per line (default of 80 usually). There are two ways of dealing with those—one is to wrap the text to the following lines(s), and the other is to “pan-and-scan”—let the text disappear off the screen and pan left-and-right to show the longer lines. Each method requires chopping text to fit though. With ASCII, this is trivial—if the width of the temrinal is N columns wide, just chop the line at every N-bytes. This works because each character in ASCII is one byte in size. But characters in UTF-8 take a variable number of bytes, so chopping at arbitrary byte boundaries is less than optimum.

The solution may look simple:

-- ************************************************************************
-- usage:       writeslice(left,right,s)
-- descr:       Write a portion of a UTF-8 encoded string
-- input:       left (integer) starting column (left edge) of display screen
--              right (integer) ending column (right edge) of display screen
--              s (string) string to display
-- ************************************************************************

local function writeslice(left,right,s)
  local l = utf8.offset(s,left)      or #s + 1
  local r = utf8.offset(s,right + 1) or #s + 1
  tty.write(s:sub(l,r - 1))

but simple is never easy to achieve. It took a rather surprising amount of time to come up with that solution.

The other major problem was dealing with the gopher index files. Yes, they are easy to parse (once you wrap your head around some of the crap that presents itself as a “gopher index” file) but displaying it was an even harder problem.

Upon loading a gopher index, I wanted the first link to be highlighted, and use the Up and Down keys to select the link and then the enter key to select a page to download and view. Okay, but not all lines in a gopher index are actual links. In fact, there are gopher index files that have no actual links (that surprised me!). And how do I deal with lines longer than can be displayed? Wrap the text? Let the text run off the screen?

At first, I wanted to wrap long lines, but then trying to manage highlighting a link that spans several lines when it might not all be visible on the screen (the following lines might be off the bottom, for instance) just proved too troublesome to deal with. I finally just decided to let long lines of text run off the end of the screen just to make it easier to highlight the “current selection.” Also, most gopher index pages I've come across in the wild generally contain short lines, so it's not that much of a real issue (and I can “pan-and-scan” such a page anyway).

For non-text related files, I farm that out to other programs via the mailcap facility found on Unix systems. That was an interesting challenge I will probably address at some point.

There are still a few issues I need to address, but what I do have works. And even though it's written in Lua it's fast. More important, I have features that make sense for me and I don't have to slog through some other codebase trying to add an esoteric feature.

And frankly, I find it fun.

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2019 by Sean Conner. All Rights Reserved.