The Boston Diaries

Saturday, February 24, 2001

EMAIL IS NOT INSTANT MESSAGING!

Reading my email at work, I see this one from E, my boss:

[Out department] just observed a momentary connectivity problem with the [city 1] data center. This was caused by the [city 2] circuit bouncing and traffic being momentarily routed through [city 3]. The circuit restored itself and appears normal at this time, however the [primary] NOC has opened [a trouble ticket] and is monitoring this circuit for additional problems.

I don't know why this upset me so much. I mean, the other email I got tonight was much worse than this (again, critical of the work done on third shift, like what else is new) but I think it has something to do with a misconception of just how the Internet works.

The closest city mentioned is well over 600 miles away, on The Company backbone (technically, we're on The Company backbone, but being way down here in South Florida, it's more of a spur than a backbone. A fast spur yes, but a spur nonetheless). And on a network the size we have here (even the network here in Boca Raton is a site to see and I don't think it's the largest one The Company has) it's expected there to be some transitory glitches. I mean, why else have redundant routes?

I think what bugs me is that most of our clients (heck, even our fellow employees) don't realize that TCP/IP is an unreliable (read: best effort) protocol and there are no guarentees on delivery of packets. And that if the routers are configured correctly (and we have some pretty sharp network engineers working here—I mean, The Company does run a backbone here) that one router doing down won't effect the connectivity (since there are redundant routes throughout the routing mesh). Okay, it may take a few minutes, but that's to be expected as routing tables restabalize.

Let me say that again: it may take a few minutes!

Like email delivery. We get calls from customers complaining that it's taking fifteen minutes for email to get through. Thankfully I'm not on the phones least I be tempted to tell these people that email is not an instant messaging protocol and that it's faster than the alternative, the snail mail postal service. Depending upon network congestion, it may take several hours for email to get through.

I'm fond of saying “Email is not FTP.” I may have to amend that to be: “Email is not FTP nor IM!”

This is also related to an incident that happened a few days ago. Just as 1^st shift was coming in, we lost connectivity with a machine we manage over in Europe. Doing a traceroute showed the loss of connectivity happening within Cable and Wireless, about three hops into their network. My thought—okay, it's not us, just inform the techs that we can't reach the machine in question and it's outside our hands.

Yet JM, 1^st shift worker, came in, and opened a trouble ticket with our primary NOC, even though the outtage was in a different backbone several hops inside. For some reason that bothered me too, I think for similar reasons. Given the amount of email we get about outtages of routers all along our backbone you'd think our networking staff would have enough work just on our equipment; why bother adding to the case load about other companies' equipment? But alas, JM did it anway, as part of the CYA attitude around here.

Sigh.

More grumbles

Along those lines, a webserver yesturday had a very bad hard drive crash and The Company has been having some difficulty in doing a proper restore (having to rebuild an entire server, and primarily doing a network based recover when a (poorly announced, but announced nonetheless) router reboot caused it to fail part way through (or so I'm lead to believe), so it's taking some time to get everything back.

But clients are complaining about The Company loosing some, but not all, the files. And complainint bitterly. Okay, fine. We had a hardware failure. Whole server blown out of the water. Recovery taking some time, and yes, there was a window where files may have been lost between the last good backup and the failure, if any files were updated during that window.

But come on—you have a hosted website and you (or the company that did your website) don't have a current backup yourself? What's this you say? The hosting company is responsible for backups? They say they backup? Here, let me fill you in on something: THWACK! Don't ever count on the hosting company to keep current backups. It would be better (and probably faster recovery) for you to keep a backup of your site and restore it yourself. Even if the hosting company keeps good backups, they're usually so large it takes several hours to recover the data to begin with.

Microsoft rethink

Our computers are big and powerful now, and at AAAS I came to the conclusion that Moore's First Law has about 50 years to run, while there are ways to circumvent Moore's Second Law (this will be in my reports). That argues enough computer power to make anything run fast. The cycles and memory will be out there. Perhaps it is time to START OVER with OS. Heck, I'd like to see Niklaus Wirth's Oberon and Modula 2 come back. I would even be interested in an OS that was written in and ran ADA as its assembly language. We are no longer constrained by memory and and cycles limits. I think it is time to break free from not only Microsoft but unintelligible code like C (which means we break free of LINUX and the other UNIX type systems too).

Jerry Pournelle on Microsoft and Linux

The page will move in a few days, so if the above link doesn't work, try this one.

Wow. I really didn't expect Jerry Pournelle to come out and say this, seeing how he tends to get personal tech support from CEOs and is an active Microsoft supporter. The times, they are a'changing.

Following the scent

Leaving work I ran into one of the techs and we started chatting a bit about the problems with the webserver we were having. I gave my rant about the clients.

“I know, and you know,” he said, “that they have a backup copy of the site on their development machines.”

“But why don't they just upload the copy they have then?”

“Because they can bitch at us and demand money back for lost services.”

“So their business isn't important enough to get the website back up and running quickly?”

“Webservers are still unreliable. It's expected they'll go down. Give it several more years, but right now, it's expected they'll go down.”

“Uh … ”

“And further more,” he said, pausing to take another drag on the cigarette he was smoking, “the dealers make out.” The server in question is used mostly be resellers—we don't deal with the owners of the sites in question, the dealers do. “They'll bitch at us for a refund, but do you think they'll refund their customers' money?” Another drag. “No, they'll keep billing 'em! Pocket the money.”

“Oh.”

“You know, The Company is going to loose a lot of money over this.”

I had forgotten the cardinal sin learned in the 70s: Follow The Money!

Why I haven't heard of it

I'm still not convinced. I wrote a quick program to act as a webserver (it serves up the same page reguardless of what you request) and had it log the request the client sends it. I got this from the most widely used browser:

GET /~spc/bm/ HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)
Host: linus.slab.conman.org
Connection: Keep-Alive

Nothing about screen size or resolution as far as I can see.

God, are we lazy!

Transcript of an AIM session between my roommate Rob and myself:

azagy214: so where do you want to go for lunch?

spc476: Hmmm …

spc476: Restaurant?

azagy214: any particular one?

spc476: One with edible food?

azagy214: care to narrow it done more?

azagy214: oopps done = down

spc476:(oh, check out http://XXXXXXXXXXXXXXXXXXXXX/~spc/webcam/ 8-)

spc476: Diner?

azagy214: cute

azagy214: diner works for me

spc476: Okay. When?

azagy214: now?

spc476: Sure.

spc476: I'll be over there in a few seconds …

azagy214: ok =)

I should mention that we were about twenty feet apart.

Heh … why not?

Ring.

“Hello?”

“Hey Sean. It's Jeff.” It's my friend JeffC. “Can I ask a favor?”

“Sure. What's up?”

“Sarah and I are moving,” he said. Sarah is his fiancée. “And we need some help.” D'oh! “You only have to help unload the truck.”

I haven't been to sleep since 9:00 pm yesturday. It is now 3:00 pm. I'm still wide awake, and for the first time since October, I don't have to work tonight. Hey, how bad could it be? “Sure. Get me get dressed and I'll be right over.”

Famous last words

“See you later, Rob.” I said.

“What's up?”

“I'm off to help a friend move,” I said. “I only have to help unload the truck.”

“Watch it! They could be moving to a third floor apartment,” he said.

“Yea … right. See you later.”

Twenty-eight stairs

A bazillion boxes.

Twenty-eight stairs.

Three floors.

Two landings.

And one full truck.

Yea, right, indeed.

Plumbing lessons

My friend Kurt (same one I went haunted house hunting with last year) arrived to help help us movie. He had just finished a class in checking back-flow prevention valves, which is actually more interesting than it sounds.

Kurt is currently a high school English teacher who is burned out—long hours a poor pay have taken their toll on him, so he's getting into the family business of plumbing. Only it's not plumbing as in plumbing repair, it's more like plumbing engineering.

Between hauling boxes and furniture up to the apartment he would stop and explain how back-flow prevention valves work. Basically there are three chambers. The first one is at a high pressure, say, 40 PSI. It hits a plate separating the first and second chambers and that plate has a spring holding it in place, but the spring has a much less pressure, say, 5 PSI. That means the water from the first chamber enters the second chamber but at a reduced pressure, which is the difference between the water presure of the first chamber minus the pressure of the plate; in this example, the water in the second chamber is now 35 PSI. There's another plate between the second and third chambers, and this one supplies even less pressure, say 1 PSI, so the water in the final chamber will only be 34 PSI.

Now, in the bottom of the second chamber is a diaphram, and water from the first chamber is directly allowed to fill the area below the diaphram. This pushes the diaphram up, because the pressure above is lower than below (35 PSI above, 40 below). Now, if the presure in the third chamber rises, it shuts the plate, causing the pressure to rise in the second chamber. This pushes the diaphram down, opening a valve to release the water directly out of the system instead of letting it build up into the system.

Quite interesting. And you have to be careful in checking some of these back-flow valves, especially the larger ones, as they can quite literally explode in your face if you're not careful.

Now a word from our sponsor …

Price of truck rental:	$39.95
Price of pizza for friends:	$40.00
Price of dolly from back of truck:	$5.00
Price of passing out on floor from sheer exhaustion:	Priceless