Saturday, January 12, 2019
The technical differences between HTTP and gopher
… The point is to attempt as full a sketch as possible of the actual differences and similarities between the HTTP and GOPHER protocols.
…
From what I gather, these are the similaries:
- Both gopher and http start with a TCP connection on an IANA registerd port number.
- Both servers wait for text (the request) terminating in a CRLF
- Both servers expect the request (if there is one) to be formatted in a particular way.
- Both servers return plain text in response, and close the TCP connection.
And these are the differences that I understand:
- Gopher will accept and respond to a blank request, with a default set of information, http will not.
- Gophper [sic] sends a single "." on a line by itself to tell the client it is done, http does nothing similar prior to closing the connection.
- Http has things like frames, multiplexing, compression, and security; gopher does not.
- Http has rich, well-developed semantics, gopher has basic, minimalist semantics
- Http requests are more resource intensive than gopher requests.
- Http is highly commercialized, gopher is barely commercialized.
- Http is heavily used and highly targeted by malicious users, gopher is neither.
- Http is largely public, gopher is largely private (de facto privacy through obscurity.)
- Http is used by everyone, their children, their pets, their appliances, their phones, and their wristwatches; gopher is used primarily by technical folk and other patient people.
- Http all but guarantees a loss of privacy; gopher doesn't
Yeah, I know, it's not much, but that's all that is coming to mind presently. What are your thoughts?
Technology/Gopher(I'm quoting for the benefit of those that cannot view gopher based sites).
I don't want to say that tfurrows is wrong,
but there is quite a bit that needs some clarification,
and as someone who has worked with HTTP for over twenty years,
and has recently dived back into gopher
(I used it for several years in the early 90s—in fact,
I recall Time Magazine having a gopher server back then)
I think I can answer this.
First, the protocol. The gopher protcol is simple—you make a TCP connection to the given port (defaults to 70). Upon connection, the client then sends the request which can be one of three formats:
CRLF
The simplest request—just a carriage return and line feed character. This will return the main page for the gopher server.
selector-to-viewCRLF
This will return the requested data from the gopher server. The specification calls this a “selector.” And yes, it can contain any non-control characters, including spaces. It's terminated by a carriage return and line feed characters.
selector-for-searchHTsearch terms to useCRLF
The last one—this sends a search query to a gopher server. It's the “selector” that initiates a search, followed by a horizontal tab character, then the text making up the query, followed by a carriage return and line feed.
In all three cases, the gopher server will immedately start serving up the data. Text files and gopher indexes will usually end with a period on its own line; other file transfers will end with the server closing the connection.
That's pretty much the gopher protocol.
The HTTP protocol that works the closest to gopher is the so called HTTP/0.9 version, and it was pretty much the the same. So the same three requests above as HTTP requests.
GET /CRLF
The minimum request for HTTP.
As you can see,
it's only an extra four characters,
but the initial text,
GET in this case,
was useful later when the types of requests increased
(but I'm getting ahead of myself here).
This will return the main page for the HTTP server.
GET /resource_to_viewCRLF
The usual request,
but instead of a “selector” you request a “resource”
(different name, same concept)
but it cannot contain bare spaces—they have to be encoded as %20
(and a bare “%” sign is encoded as %25).
Like gopher,
the contents are immediately sent,
but there is no special “end-of-file” marker—the server will just close the connection.
GET /resource_for_seach?search%20terms%20to%20useCRLF
And a search query,
where you can see the spaces being replaced with %20.
Also note that the search query is separated by the “resource” with a “?”.
So not much difference between gopher and HTTP/0.9. In fact, during the early to mid-90s, you could get gopher servers that responded to HTTP/0.9 style requests as the difference between the two was easy to distinguish.
The next version of HTTP, HTTP/1.0, expanded the protocol. Now, the client was expected to send a bit more infomration in the form of headers after the request line. And in order to help distinguish between HTTP/0.9 and HTTP/1.0, the request line was slightly expanded. So now the request would look like:
GET /resource_to_view HTTP/1.0CRLF User-Agent: Foobar/1.0 (could be a web browser, could be a web crawler)CRLF Accept: text/*, image/*CRLF Accept-Language: en-US;q=1.0, en;q=0.7; de;q=0.2, se;q=0.1CRLF Referer: http://www.example.net/search?for%20blahCRLF CRLF
(Yes, “Referer” is the proper name of that header, and yes, it's mispelled)
I won't go too much into the protocol here,
but note that the client can now send a bunch more information about the request.
The Accept header now allows for so-called “content negotiation” where the client informs the server about what type of data it can deal with;
the Accept-Language header tells the server the preferred languages
(the example above says I can deal with German, but only if English isn't available,
but if English is availble,
American is preferred).
There are other headers;
check the specification for details).
The server now returns more information as well:
HTTP/1.0 200 OkayCRLF Date: Sun, 12 Jan 2019 13:39:07 GMTCRLF Server: Barfoo/1.0 (on some operating system, on some computer, somewhere)CRLF Last-Modified: Tue, 05 Sep 2017 02:59:41 GMTCRLF Content-Type: text/html; charset=UTF-8CRLF Content-Length: 3351CRLF CRLF content for another 3,351 bytes
The first line is the status,
and it informs the client if the “resource” exists
(in this case, a 200 indicates that it does),
or if it can't be found
(the dreaded 404)
or if it has explicitely been remove
(410)
or it's been censored due to laws
(451),
or even moved elsewhere.
Also added were a few more commands in addition to GET,
like POST
(which is used to send data from the client to the server)
and HEAD
(which is like GET but doesn't return any content—this can be used to see if a resource has changed).
HTTP/1.1 is just more of the same, only now you can make multiple requests per connection, a few more commands were added, and the ability to request portions of a file (say, to resume a download that was cut off for some reason).
HTTP/2.0 changes the protocol from text-based to binary (and attempts to do TCP-over-TCP but that's a rant for another time) but again, it's not much different, conceptually, than HTTP/1.1.
Security,
as in https: type of security,
isn't inherently part of HTTP.
TLS is basically inserted between the TCP and HTTP layers.
So the same could be done for gopher—just insert TLS between TCP and gopher and there you go—gophers:.
Of course,
that now means dealing with CAs and certificates and revocation lists and all that crap,
but it's largely orthogonal to the protocols themselves.
HTTP/1.0 allows compression but that falls out of the content negotiation. The bit about frames and multiplexing is more an HTTP/2.0 issue which is a lot of crap that the server has to handle instead of the operating system (must not rant …).
Are HTTP requests more resource intensive? They can be, but they don't have to be. But that leads right into the commericalization of HTTP. Or rather, the web. HTTP is the conduit. And conduits can carry both water and waste. HTTP became commercialized because it became popular. Why did HTTP become popular and gopher whithered? Personally, I think it has to do with HTML. Once you could inline images inside an HTML document, it was all over for gopher. The ability to include cat pictures killed gopher.
But in an alternative universe,
where HTML had no image support,
I think you would have seen gopher expand much like HTTP has.
Work was started in 1993 to to expand the gopher protocol
(alternative link)
where the protocol gets a bit more complex and HTTP-like.
As mentioned,
a secure gophers: is “easy” to add in that it doesn't change the core protocol.
And as such,
I could see it getting more commercialized.
Advertising can be inserted
TYPEWRITERS
For SALE, HIRE, or EXCHANGE,
at HALF the USUAL PRICES.
MS. Typewritten from
10d. per 1,000 words.
100 Circulars for 4s
TAYLOR'S,
74, Chancery Lane, London.
(Est. 1884.)
Telegrams: "Glossator," London.
Telephone No. 690, Holborn.
even in a text file. Yes, it might look a bit strange, but it can be done. The only reason it hasn't is that gopher lost out to HTTP.
So those are the differences between HTTP and gopher. HTTP is more flexible but more complex to implement. Had history played out differently, perhaps gopher would have become more flexible and complex.
Who knows?
It's no longer possible to write a web browser from scratch, but it is possible to write a gopher browser from scratch
As I mentioned two months ago, I've been browsing gopherspace. At the time, I was using an extension to Firefox to browse gopherspace, but a recent upgrade to Firefox left it non-working. I could use Lynx but it includes unnecessary pauses that make it feel slower than it really should be. I also don't care for how it lays out the page.
So I've been writing my own CLI gopher client.
And it's not like the protocol is all that difficult to handle and everything is plain text.
How hard could it be to download a bunch of text files and display them?
The protocol? Trivial.
Displaying the pages? Er … not so trivial.
The first major problem—dealing with UTF-8.
Problem—the terminal window can only display so many characters per line
(default of 80 usually).
There are two ways of dealing with those—one is to wrap the text to the following lines(s),
and the other is to “pan-and-scan”—let the text disappear off the screen and pan left-and-right to show the longer lines.
Each method requires chopping text to fit though.
With ASCII, this is trivial—if the width of the temrinal is N columns wide,
just chop the line at every N-bytes.
This works because each character in ASCII is one byte in size.
But characters in UTF-8 take a variable number of bytes,
so chopping at arbitrary byte boundaries is less than optimum.
The solution may look simple:
-- ************************************************************************ -- usage: writeslice(left,right,s) -- descr: Write a portion of a UTF-8 encoded string -- input: left (integer) starting column (left edge) of display screen -- right (integer) ending column (right edge) of display screen -- s (string) string to display -- ************************************************************************ local function writeslice(left,right,s) local l = utf8.offset(s,left) or #s + 1 local r = utf8.offset(s,right + 1) or #s + 1 tty.write(s:sub(l,r - 1)) end
but simple is never easy to achieve. It took a rather surprising amount of time to come up with that solution.
The other major problem was dealing with the gopher index files. Yes, they are easy to parse (once you wrap your head around some of the crap that presents itself as a “gopher index” file) but displaying it was an even harder problem.
Upon loading a gopher index,
I wanted the first link to be highlighted,
and use the Up and Down keys to select the link and then the enter key to select a page to download and view.
Okay,
but not all lines in a gopher index are actual links.
In fact,
there are gopher index files that have no actual links
(that surprised me!).
And how do I deal with lines longer than can be displayed?
Wrap the text?
Let the text run off the screen?
At first, I wanted to wrap long lines, but then trying to manage highlighting a link that spans several lines when it might not all be visible on the screen (the following lines might be off the bottom, for instance) just proved too troublesome to deal with. I finally just decided to let long lines of text run off the end of the screen just to make it easier to highlight the “current selection.” Also, most gopher index pages I've come across in the wild generally contain short lines, so it's not that much of a real issue (and I can “pan-and-scan” such a page anyway).
For non-text related files,
I farm that out to other programs via the mailcap facility found on Unix systems.
That was an interesting challenge I will probably address at some point.
There are still a few issues I need to address, but what I do have works. And even though it's written in Lua it's fast. More important, I have features that make sense for me and I don't have to slog through some other codebase trying to add an esoteric feature.
And frankly, I find it fun.
Tuesday, January 08, 2019
A reason to celebrate today
The book Atlas Obscura: An Explorer's Guide to the World's Hidden Wonders appeared on the doorstep, courtesy of Bunny. I do have to wonder why she felt it necessary to give me that book to celebrate the only time in the history of the United States the country was not in debt, back in 1835 thanks to President Andrew “I regret not killing that bastard Vice President Calhoun” Jackson. It could be for other reasons, but I can't be sure.
In any case, it's a very cool book. Where else could one learn about the Skunk Ape Research Headquarters in Ochopee, Florida? Or the Lost Subway of Cincinnati, Ohio? Lots of places to visit—oh!
Monday, January 07, 2019
Ignorance is Bliss
So I'm catching up on The Transylvania Times (I'm a bit behind, and they're piling up) when I come across this rather distrubing headline: “Wednesday Morning Earthquake Felt In Transylvania.”
Wait … what?
Yes Virgina, an earthquake in the southeast United States.
I know eathquakes happen along the Pacific Coast (like California, the land of Shake and Bake). I also know they happened in Missouri (although rare, when it happens, it happens). But in the East? The East is supposed to be stable. Rock solid (ahem). Not shifting underneath our very feet. I am disquieted by this news.
As I fall deeper into this whole “East Coast Earthquake Zone,” it appears to be all too true. There's a fault line that runs from Alabama northeast to Newfoundland, Canada, and it runs about six miles east of Brevard.
I … I don't know how I feel about this. I admit, I have an irrational fear of earthquakes. I don't know why, I just do. Hurricanes? Please … don't bother me unless it's a category 4. An earthquake? Even a relatively minor 2 on the Richter scale (and this one was a 4.4)? Aieeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee! Run away!
And the kicker in all this? This fault line has a name. It's name is the Brevard Fault! No, really.
Now I really have no idea how I feel about this.
Tuesday, January 01, 2019
Yes, we have no copyright
I just saw a commerical using the Prince song “Let's Go Crazy”. It was something I wasn't expecting because Prince had refused all requests to use his work for commericals (as well as turning down all requests from Weird Al Yankovic to parody his songs). But given that Prince died back in 2016 it seems his estate has waited long enough and is now enjoying the licensing fees.
Then it hit me—it won't be until 2086 that the works of Prince will fall into the public domain. Nearly a hundred years since some of his most iconic hits.
In other copyright-public-domain news, today is the first day in 21 years that works have fallen into the public domain. It's weird to think that up until yesterday, “Yes, We Have No Bananas” was still in copyright.
The upside is that this is not an election year—the downside is that we only have 364 days until it is
I've been looking over the past decade's worth of New Year's Day entries so I don't inadvertantly repeat myself, and boy, do I bitch about the fireworks. Thankfully this holiday season has been a bit low key, and that includes our neighbor's propensity towards blowing things up. Yes, there were fireworks tonight, but not nearly at the war sounding levels of previous years.
I'll take solace when I can when it comes to fireworks.
Hopefully, this means that this year will be low key. All I can say is thank God it isn't an election year! Only 364 more days until that madness starts.
Anyway …
HAPPY NEW YEAR!
Wednesday, Debtember 26, 2018
Observations made at a mall one day after Christmas
Had I known the traffic around the Town Center Mall was going to be so insane, I would have suggested a different location to meet Bunny. Parking was insane along the Glades Road side—cars were parked everywhere! Even on the grass, and that's saying something because of the 6″ curb between the asphalt the grass! The normal traffic was changed for the holidays and I ended up back on Glades Road unexpectedly. I ended up parking in the back 40 where parking wasn't quite as insane.
I had no idea Microsoft had physical stores, much less one at the mall! I guess they have to keep up with Apple, although the Microsoft store is easily a sixth the size of the Apple one. It's kind of sad, and understandable why it's located away from the Apple store.
I have to wonder about the wisdom of letting high school girls play Just Dance 2015 in the middle of the mall. I also have to wonder about the wisdom of the high school girls who were dancing the Macarena in the middle of the mall. Really? That over “You Spin Me Round (Like a Record)”?
Kids today.
I was so tempted by the Häagen-Dazs store. But I made my saving throw vs. ice cream and resisted the temptation. I thought I was safe until I saw the Godiva store selling ice cream. I was seriously tempted, but again, I was just able to make my savings throw vs. ice cream.
When did Godiva start selling ice cream? And why did I not get the memo?
Ah, Gallerit! How can I describe you? Um … nouveau riche tackiness. Human sized bronze statues of half-naked women. Gemstone globes on gilted ostentatious stands. Giant sized bronze statues of Brittish Bull Dogs. Famous painting reproductions in huge gaudy frames.
All of it horribly tacky! All of it glorious to behold! And much larger than the Microsoft store!
Tuesday, Debtember 25, 2018
Printing beyond the bounds of two dimensions
I'm not a fan of printers.
My first printer was a daisy wheel printer that sounded like a machine gun when working. I only had two “wheels” (a cartridge with the character set splayed out in a wheel shape, hence the name), I think both where Courier, but one was 10 characters per inch, the other one 12 characters per inch. Despite the price, I think I used it for one term paper (high school) and for the humor column for the university paper (that is, when I wasn't frantically pounding out an article on my portable manual typewriter 20 minutes prior to the deadline, but that's another story). It was obnoxiously loud (Harley owners would complain of the noise) and obnoxiously large.
My second printer was an 80-column dot matrix that used fan-fold paper. It was quieter than my first printer (what wouldn't be?), a bit faster, and much less fuss as I didn't have to constantly feed paper into the thing. I used it a bit more than my first printer, but as time went on, I found myself using it less and less.
Printers ever since have been owned by significant others. Cheaper to buy, yes. More expensive to maintain as printer companies learned to charge outrageous sums for inks and toner cartridges and after a few scant years they just stop working. Horrible things in my opinion (I think we're on our fourth such printer at Chez Boca since I moved in over a decade ago). The only time I use them these days is to print out the tax forms once a year (and for those, I could go down to the library and pick up the proper forms).
So guess what I got for Christmas?
A printer.
But unlike the printers of the past, this is a freaking 3D printer!
Woo hoo!
I can print solid objects now!
My only complaint about the printer is the manual. Upon turning it on, it made a horrible noise (much like the sound of the Apple ][ disk drive as the head was moved to a known location) and the small screen on the unit gave an error number. The manual listed the symptom as “Y-axis movement abnormalities” and the action “check motor/sensor connection.” That's it! Nothing else about how to go about checking the motor or the sensor.
Sigh.
It turned out to be an easy fix (just force the “print head” forward all the way in the case—it had slipped a bit in shipping) and a few minutes later I was printing my first object (a plastic heart pendant for Bunny). It took about as long as expected (half an hour) and was a bit harder to remove from the printing platform than expected, but it works.
And judging by this video, it could prove to be useful to have around the workshop.
Something something peas on the hearth … something someting wind towards Mars, I don't know … there's something significant about today
* ,
_/^\_
< >
* /.-.\ *
* `/&\` *
,@.*;@,
/_o.I %_\ *
* (`'--:o(_@;
/`;--.,__ `') *
;@`o % O,*`'`&\
* (`'--)_@ ;o %'()\ *
/`;--._`''--._O'@;
/&*,()~o`;-.,_ `""`)
* /`,@ ;+& () o*`;-';\
(`""--.,_0 +% @' &()\
/-.,_ ``''--....-'`) *
* /@%;o`:;'--,.__ __.'\
;*,&(); @ % &^;~`"`o;@(); *
/(); o^~; & ().o@*&`;&%O\
jgs `"="==""==,,,.,="=="==="`
__.----.(\-''#####---...___...-----._
'` \)_`"""""`
.--' ')
o( )_-\
Thursday, Debtember 13, 2018
This reminder is to remind you of the reminder
So I received this in email today:
- From
- Chevrolet <XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
- To
- sean@conman.org
- Subject
- Sean, employee pricing is back on most Chevy cars, trucks and SUVs
- Date
- Thu, 13 Dec 2018 17:11:46 -0800 (PST)
This is plain text, with a line break:
Next new line
And the thing that I immediately thought of were old IBM manuals that would have a page with the following text:
This page intentionally left blank.
No it's not. It should have read:
This page intentionally left blank, except for this line of text.
At least this email from Chevrolet is more accurate than the old IBM manuals. But then again, I wonder what Chevrolet was trying to sell me.
![[It's the most wonderful time of the year!]](http://www.conman.org/people/spc/about/2018/1212.t.jpg "It's the most wonderful time of the year!")