The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Sunday, January 01, 2006

Happy New Year

At some arbitrary point in time we slip into a new year and thus ends the obligatory “See you next year!” jokes for approximately 360 days.

We also, at this arbitrary point in time, make promises to ourselves to improve this, change that, and to stop doing the other. Such self dellusion usually ends in another 40 days or so as time marches on.

We also hope that the following 365 days, six hours, mine minutes and nine seconds (or 365 days, five hours, 48 minutes and 45 seconds, or 365 days, six hours, 13 minutes and 52 seconds, depending upon how one defines the term “year”) will be much better than the previous 365 days, six hours, mine minutes and nine seconds (or 365 days, five hours, 48 minutes and 45 seconds, or 365 days, six hours, 13 minutes and 52 seconds, depending upon how one defines the term “year”).

So much hopes and dreams and schemes pinned on the flipping of a single second at some arbitrary point in time.

Anyway …


Monday, January 02, 2006

Tar pits, legal and otherwise

Now that I more or less run my own DSL connection, I'm able to have more than a single static IP address. Because of that, I've given my main home system a public IP address so that I no longer have to log into my firewall/NAT system to get to it.

One thing about sitting next to a server is that over time, you get used to the various patterns of noise the harddrive makes. It starts quickly grinding, I know that a load of spam is coming in via my backup email server (a common tactic of spammers—send spam to the backup servers which have less information about valid users so they can “honestly” say they delivered the email). The grinding that starts at 4:00 am are various cron jobs rotating logs, clearing out temporary files and what not.

But over the holiday weekend linus (the machine in question) started making some disk noises I'm unfamiliar with. A constant “click click” noise, perhaps a second apart. I start poking around the system and it's a scripted attack, attempting to log into my system:

Jan  1 13:57:50 linus sshd[24760]: Failed password for root from XXX.XXX.XXX.198 port 43440 ssh2
Jan  1 13:58:05 linus sshd[24784]: Failed password for nobody from XXX.XXX.XXX.198 port 45220 ssh2
Jan  1 13:58:06 linus sshd[24786]: Failed password for root from XXX.XXX.XXX.198 port 45351 ssh2
Jan  1 13:58:14 linus sshd[24799]: Failed password for www from XXX.XXX.XXX.198 port 46207 ssh2
Jan  1 13:58:27 linus sshd[24819]: Failed password for news from XXX.XXX.XXX.198 port 47673 ssh2
Jan  1 13:58:29 linus sshd[24823]: Failed password for games from XXX.XXX.XXX.198 port 47977 ssh2
Jan  1 13:58:33 linus sshd[24829]: Failed password for mail from XXX.XXX.XXX.198 port 48413 ssh2
Jan  1 13:58:34 linus sshd[24831]: Failed password for adm from XXX.XXX.XXX.198 port 48563 ssh2
Jan  1 13:59:03 linus sshd[24877]: Failed password for operator from XXX.XXX.XXX.198 port 51355 ssh2
Jan  1 13:59:12 linus sshd[24891]: Failed password for sshd from XXX.XXX.XXX.198 port 51947 ssh2

Hundreds of attempts.

Now, I could block them on the sever:

GenericUnitRootPrompt# route add -host XXX.XXX.XXX.198 reject

But that would only block them from my one server. And besides, there's this firewall between the internet and the DSL router at the office. Cut off the traffic a bit upstream so they can't scan any of the DSL connections:

GenericUnixRootPrompt# iptables --table filter --insert FORWARD --source XXX.XXX.XXX.198 --jump REJECT

And boom, the traffic stops dead.

For a few hours.

“Click click click click”

Yet another brute force attempt to log into my server. Previous one was from Germany. This one from the Netherlands. Another one a few hours later from Korea. One from Lakeland, Florida!

I ended up stopping eight of these attempts over the weekend—not because my system was particularly vulnerable, but the “click click click” of the disk drive was rather annoying (I should note that the drive itself is not going bad—it's just the sounds the heads make when seeking across the platters).

“Click click click click”

In fact, there's an attempt right now going on as I type this, from Korea—or rather, there was an attack.

And the servers at The Office get scanned constantly. I should know, I get the multithousand line log summaries daily.

So with all that in mind, I'm trying to cobble up a LeBrea Tarpit system at The Office. Legal disclaimers on the site aside, it's still available and in checking the rather loathsome Florida's Super DMCA Act, I think legally we can run it—we're not a “cable operator” (¶ (1)(a) §1 §812.15, F.S.), nor are we a “communications service provider” under the definition given (¶ (1)(e) §1 §812.15, F.S.), nor do we provide a “communications service” under the given definition (¶ (1)(d) §1 §812.15, F.S.). And even if we are classified as a “cable operator” (and really, the definition screams “television broadcasting”), then it falls under ¶ (2)(a) §1 §812.15, F.S.:

(2)(a) A person may not knowingly intercept, receive, decrypt, disrupt, transmit, retransmit, or acquire access to any communications service without the express authorization of the cable operator or other communications service provider, as stated in a contract or otherwise, with the intent to defraud the cable operator or communications service provider, or to knowingly assist others in doing those acts with the intent to defraud the cable operator or other communications provider.


See … we are the communications service provider in that case. We're not out to defraud ourselves! We gave ourselves permission!

And it's not entrapment. LeBrea isn't simulated a poorly administered system just waiting to be hacked. It just accepts TCP connections and keeps them on hold (as it were) indefinitely. It's not violating the letter of the TCP specification (it could be argued it's violating the spirit of the TCP specification, but really, what jury will want to spend time listening to geeks pontificate about the finer points of dogmatic religious theology TCP connection semantics?

Tuesday, January 03, 2006

“Little orange houses, for you and me … ”

For Spring, the Orange Houses of Detroit (link via dETROITfUNK).

Yeah … I'm walking kind of funny now …

Due to Hurricane Wilma, then the holiday season, we ended up being just a wee bit behind in our electrical bill. In discussing this last night, it was decided that I would pay the outstanding amount on the way to work today. There's a gas station about a mile or so that one can pay their utility bills. I was expecting this to take only a few minutes or so.

I arrive at said gas station and I'm the only customer. I hand over the bill to the cashier who starts running it through some device (to read the account and amount off the numbers written in magnetic ink along the bottom), then punching lots of buttons, then running it through again, then punching numbers, then marking up the bill with incompresensible hieroglyphics, running it through again, then violently jabbing a pin-infested cloth doll; pins are falling out and bouncing off the counter top. Minutes go by, and the cashier finally turns to me.

“I'm sorry,” the cashier said. “But the bill is being rejected.”

“Rejected?” Oh XXXX, I thought. They've already cut our account and are releasing the hounds thugs collection agents as I stand here.

“Yes, we always have problems with Lake Worth Utilities,” said the cashier, glancing towards the cloth doll. “No problems with any other company. But Lake Worth Utilties.”

Lake Worthless Utilties?” I said.

“Pretty much.”


“You can pay at the City Hall. Just down Lake Worth Avenue. Down by Dixie Highway.”

“Which side of the street?”

“North side. You can't miss it.”


Couldn't miss it because the traffic lined up down the block in front of the Lake Worth City Hall Annex, waiting paitently to pay their bills in the drive-through.

Not wanting to bother with that, I park across the street, and head into the Lake Worth Cith Hall Annex building.

I don't know that it is about government buildings, but every single one is painted in Institutional Paint. It's not a single particular shade of color—sometimes a pale green, other times a jaundiced yellow, or a pale flesh tone—but the shades themselves are never bright, always dull, like it's been there forever, and will remain there forever, eternally dingy. The Lake Worth Cith Hall Annex is a pale flesh color (maybe even a sickly peach color) headed towards a dull beige.

Soul sucking, is what it is.

There's a main hallway that shoots through the Lake Worth City Hall Annex building, and lined up down this hall is a line of people, waiting to get into the Utilties office in the center of the building. A real life Kurgan walks down the hall, deciding he'll come back on Thursday. A cute girl ahead of me in line is wearing a shirt, “I may not be perfect, but parts of me are AWSOME” and yes, parts of her are awsome. At least it keeps my mind of the numbing wait in line.

Half an hour later, I'm actually in the Utilties office, still waiting in line. The cashier window is on the other side of the room where a single person is working. I assume there is at least one other person working the drive though, since I don't see the cashier leave the desk.

Half an hour later, I'm the next in line. At this point, I see they take credit cards, but just past, far enough that it's hard to read is a sheet of paper with the word “VeriCheck” written across the top. The sign nearby that shows that Lake Worth Utilties accepts credit cards also has the word “VeriCheck” across it. I step over to the sheet of paper, keeping my leg stretched out with my toe marking my spot in line, and scan the sheet of paper.

It's a list of fees.

It's a list of fees one will pay for the “convenience” of paying your grossly inflated Lake Worth Utility bill with your credit card (or in my case, a debit card).

What a XXXXXXX racket, I think. The “fee” to “conveniently” pay with your “credit card” is a “measly” 5%.


“Do you understand that there is a fee if you pay with your credit card,” said the cashier when it was my turn to pay.

“Not like I have any other choice,” I said. Notice how the signs giving the fee structure was posted next to the cashier, and not, say, out in the hall or near the door, when I might have been able to do something about it. Now, after an hour?

The cashier then mentions the fee. “Are you sure you want to continue paying?”

Not really, but then you're less inclined to continue our electrical supply, so here, let me bend over, I thought. What I said was, “Yes. No other choice really.”

“Splendid,” said the cashier, taking my bill and debit card.

Splendid indeed.

Now, I don't see this as a fault of capitalism. No, this is a fault of a government mandated monopoly that is nothing less than coercion. FPL, for all their faults, is at least working with economies of scale and are able to charge about half that of Lake Worth Utilties.

Although on my way out, I did warn a few people about the fees awaiting them, if they so choose to use plastic to pay. At least one guy started on about taxation without representation, so maybe there's hope yet …

Peekaboo! Oh … you're gone!

I have what may be some bad news for Mark. The club he likes (heck, I liked it too!), the one on Lake Worth Avenue near Congress, is no more. The building has been torn down, and there's new construction on the site (as I found out while driving around today). Given what I saw of the new building, unless the club is going to be two stories (which I doubt), the club is no more.

Sorry Mark.

Rant about my cell phone, Part I

As I was standing around in line, I was kicking myself for not having a pen, so I could make a few notes for the inevitable blog entry. Then I suddenly remembered: I have a cell phone! With a note pad on it!

Sure, it sucks having to type words using a numeric keypad, but it has enough programming to realize that when I hit 4475 it's most likely because I want to type out “girl” than “ggpj” (otherwise, I'd have to type 4 pause 444777555), but the “memo” feature only accepts about 40 cha

Rant about my cell phone, Part II

racters before I have to start a n

Rant about my cell phone, Part III

ew memo.

Stupid software limitations.

Kinda says it all, doesn't it?

Guys, you keep posting that same story about a serious security flaw in Windows.

Windows XP Flaw ‘Extremely Serious’

An anonymous quote at Slashdot about the bazillionth security hole in Windows wherein an image—about as inert as binary data can be—is now a vector for executable code to take over your system.

At this point, can't Microsoft be taken down (as in “corporate death penalty”) as either being completely malicious or completely incompetent? I mean, seriously—infected images?


Wednesday, January 04, 2006

I just love helpful software

Offered without comment.

Y:\>net share trickout=y:\testfolder
The resource named cannot be shared.

More help is available by typing NET HELPMSG 3736.

Y:\>net helpmsg 3736

The resource named cannot be shared.


Slowly working through a tarpit

Since we have this endless supply of Cobalt RaQs at The Office, I used one to set up a LaBrea Tarpit system. Playing around with it I noticed that LaBrea doesn't stop port scans per se, since port scanners tend to do the TCP handshake then drop the connection, whereas LeBrea will put a connection on hold indefinitely (assuming the other side keeps the connection up). It will also (as I ran it) accept connections on every single TCP port, all 65,536 of them, which is something that doesn't happen on a real server, so I may have to limit the number of ports LaBrea responds to.

Remotely recursive

While installing Linux on a new 1U server for a customer, I managed to play around with our new MegaRAC® K1, a rather neat KVM that works over TCP/IP. While it can be hooked up to a Linux system, the remote display software only works under Windows (of course, and IE—it demands IE). Annoying, but not too much of an issue, since I do have rdesktop installed on my Linux workstation at the Office.

So just because I can, at home I logged into my workstation at The Office. Since X Windows has had a remote desktop feature since it was first written way back in the mid-80s, it's easy enough for me to run rdesktop there and have it displayed on my computer at home (the Mac mini). Then, logging into the Windows system at The Office, I'm able to run the MegaRAC® K1 program to get to the console of the new server.

A remote desktop to my workstation, which has a remote desktop to the Windows box, which has a remote desktop to the new Unix server.

Just because I can (and yes, it's as slow as it sounds).

Thursday, January 05, 2006

Stupid PC tricks

I had so much fun with the MegaRAC® K1 that I hooked the device up to my workstation at The Office (and you can hook a keyboard, video and mouse up to it, and still remote desktop into the system), then from my system used rdesktop to get to the Windows box, from which I remote desktopped (um … yeah) back into my workstation.

The effect was much like pointing a video camera to a TV displaying what the video camera sees.

[My head es-plode]

Just because I can.

Slogging through the 'pit

Today was the first day I ran the Labrea tarpit on the network. I almost didn't leave the office since the results were most interesting. The first hour it ran (from 19:04:31 Eastern to 20:04:31) it “pitted” 9,309 connections from 865 unique IPs. And the ports involved:

Ports captured during a Labrea run of one hour
Port # Port description # connections
135 Microsoft-RPC service 4,996
445 Microsoft-DS Service 3,724
139 NetBIOS Session Service 295
22 Secure Shell Login 231
80 Hypertext Transport Protocol 62
6348 unassigned (possible worm?) 1

That Microsoft specific ports are at the top of the list are totally unexpected here.

I did learn a few things about LaBrea though. One, it only works on a single netblock. Unfortunately for us at The Company, we have several network blocks to worry about and that means either a few machines running this, or several instances (and given that LaBrea puts the network interface in promiscuous mode, I'm not sure how multiple instances would react with each other on the same interface) on different interfaces on one box.

Two, the network block does not have to match the network block the actual system is in, which saves an unsused IP address (ha ha).

A puzzling thing though. I got home, it was still running. I checked back a few hours later, and nothing past 20:21:17. LaBrea was still running, but either we captured all that could be captured, or something else was up.

Or down, as it turned out.

The interface that LaBrea was running on just died. I don't know if the switch doesn't like it (unlikely), the network cable is bad (could be—I did make the cable) or the interface just blew up (also a possibility). Even a reboot of the system didn't fix the problem. I'm hoping it's just the cable.

Packets from Mars

Another thing I'm seeing with the LaBrea Tarpit software—wierd ARP packets.

I've got the logging set to “verbose” (in case any problems come up) and one result is that it logs all the ARP requests that it sees, but can't answer (can't answer them because they're outside the normal netblock LaBrea is “answering” for). So I see a bunch for other public IP addresses we have. And the various private IP addresses that I'm not exactly sure what they're being used for, but hey, they're private.

And then … there are the ARP requests that shouldn't exist:

Jan  6 00:55:40 ltp : IP address not in netblock - ARP WHO-HAS   TELL
Jan  6 00:55:42 ltp : IP address not in netblock - ARP WHO-HAS   TELL
Jan  6 00:55:43 ltp : IP address not in netblock - ARP WHO-HAS TELL
Jan  6 00:55:43 ltp : IP address not in netblock - ARP WHO-HAS   TELL
Jan  6 00:55:44 ltp : IP address not in netblock - ARP WHO-HAS   TELL
Jan  6 00:55:44 ltp : IP address not in netblock - ARP WHO-HAS TELL
Jan  6 00:55:44 ltp : IP address not in netblock - ARP WHO-HAS   TELL
Jan  6 00:55:45 ltp : IP address not in netblock - ARP WHO-HAS TELL
Jan  6 00:55:47 ltp : IP address not in netblock - ARP WHO-HAS   TELL

I have no clue where these are coming from.

None, and I mean none of our addresses are from the block. That doesn't appear to be any of our upstream providers. I can traceroute to but why I'm seeing ARP requests from it is beyond me. Especially since ARP requests can't be routed! It's something local sending it out, or it's LaBrea misinterpreting the ARP packet (since it can be used for more than just IPMAC address translations).

One thing for sure, I'll have to check the code, and possibly modify LaBrea to log the MAC address so I can track this down. Then again, I have to hit the code anyway, since technically, the MAC address LaBrea uses is wrong (the “global/local” bit in the address is not set, and it should be).

Friday, January 06, 2006

More tarpit stuff

The problem ended up being the cable; nothing a little recrimping couldn't fix.

I did however, run LaBrea on the working port last night, and have a full twelve hours of data, from 00:00:00 (Eastern) to 11:59:59, and the results are rather amusing. 55,331 port connections on hold, from 1,743 unique IP addresses. And the only surprising thing is the low number of scans for SMTP.

Ports captured during a Labrea run of twelve hours
Port # Port description # connections
135 Microsoft-RPC service 30,218
445 Microsoft-DS Service 11,813
139 NetBIOS Session Service 5,934
4899 Remote Administration 2,412
80 Hypertext Transport Protocol 1,692
22 Secure Shell Login 1,190
6129 Dameware remote administration software 486
1080 W32.Mydoom.F@mm worm 404
2100 Oracle XDB FTP Services 377
4444 W32.Blaster.Worm 372
1433 Microsoft SQL Server 258
15118 Dipnet/Oddbob Worm 140
5000 Microsoft Universal Plug-n-Play 13
2745 Bagle/Beagle/Tanx viruses 10
25 Simple Mail Transport Protocol 7
47707 unknown 5

And it seems, from these results, that simply blocking the ports used by Microsoft Windows will stop 87% of these scans (and for our particular run, if I just blocked I would have stopped 35% of all the scans—that was a particularly persistent computer).

Update on Saturday, January 7th, 2006

I may not have been properly tarpitting the connections.

Saturday, January 07, 2006

Some updates on a tarpit

I played with LaBrea a lot yesterday and today. One of the things I found was that I may have not been actually “tarpitting” any connections on Thursday and early Friday. I had set the maximum bandwidth that LaBrea could use to 4 Kbps and that may have been too low to do any good. I upped the bandwidth to 32 Kbps and that's when I started seeing “Persist Activity” records in the log file. It looks like the bandwidth capped out, so since then I've upped the allowable bandwidth to 64 Kbps.

[NETWORK TARPIT bandwidth usage]

Another annoying aspect to LaBrea—it's not easy to change the MAC address it uses for tarpitting. As I mentioned, the MAC address it uses is technically incorrect, as it uses a “Global” address, not a “locally assigned” address. It also makes it difficult to run more than one instance of LaBrea on a network (and we may need to do so on our network until I can get all the traffic segmented properly).

I came up with an address to use, DE:CA:FB:AD:00:01 (yes, that spells “decaf bad”) which is distinctive enough to see in ARP tables. I found the line of code to change in labrea.h:100

#define ETH_ADDR_BOGUS  "\x00\x00\x0f\xff\xff\xff"
                                /* Bogus MAC addr used in IP capture */

Simple enough to change. But doing so made LaBrea unable to actually tarpit any connections. Changing the MAC address back to 00:00:0F:FF:FF:FF and it would work. Which means that LaBrea has hardcoded elsewhere in the code.

Found it too, in labrea_init.c:248

  static char bpf[]     = "arp or (ip and ether dst host 00:00:0F:FF:FF:FF)";

Believe me, if there was an option to select the MAC address to use for tarpitting, I would have used it. So I suppose I can now add such a feature, now that I know where to look for it.

So, now that I'm properly tarpitting connections, I've pulled stats from another twelve hour run, which should be more accurate than my last chart. With the additional information, I've been able to calculate the time each connection is “held” and again, it's interesting. This time, the twelve hours resulted in a tarpitting of 27,427 connections across 1,418 unique IPs with an average “hold time” of 48 minutes, 21 seconds (although there were 302 connections held for over 11 hours, presumedly they still are). Again, the ports being scanned hold nothing surprising, with this time 63% of the scans for Microsoft specific ports:

Ports captured during a Labrea run of twelve hours 00:00:00 to 11:59:59 on Jan 7, 2006
Port # Port description # connections
Port # Port description # connections
139 NetBIOS Session Service 7127
1433 Microsoft SQL Server 4605
80 Hypertext Transfer Protocol 4425
445 Microsoft-DS Service 4361
135 Microsoft-RPC service 1290
3306 MySQL Server 1157
443 Hypertext Transfer Protocol over SSL 978
3840 unknown 915
22 Secure Shell Login 892
3380 unknown 455
8080 Hypertext Transfer Protocol—typical alternative port 403
15118 Dipnet/Oddbob Worm 361
4899 Remote Administration 249
1080 W32.Mydoom.F@mm worm 79
25 Simple Mail Transfer Protocol 64
21 File Transfer Protocol command port 24
57 Mail Transfer Protocol (obsolete) 18
6885 Bittorrent 6
6129 Dameware remote administration software 5
34153 client-port on Red Hat Linux 9.0, Fedora Core 1, Red Hat Enterprise 3 4
1025 Microsoft-RPC service (what? Another one?) 3
18448 unknown 2
524 Novell Core Protocol request 1
20298 unknown 1
2008 Microsoft terminaldb configuration 1
18000 unknown 1

I may need to write a program to process the logging in real time, since LaBrea is quite versbose, as it looks like it'll generate about 150M per day of logging information.

Sunday, January 08, 2006

Elvis has left the planet

Last month I wrote about the Weekly World News and how I was sad to see it have a disclaimer about the fictious nature of its articles. And today, another thought hit me—one thing you no longer see in taboids are Elvis sightings (this being his 71st birthday and all). Has the tabloid world finally realized that Elvis “The King” Presley has not only left the building, but actually left this plane of existence? That scores of fans and New Age cranks come to grips that Elvis will no longer haunt midwest grocery stores and fried chicken fast food restaurants?

Say it ain't so!

Monday, January 09, 2006

The user expectation is on fire

A recurring theme here is one of user expectation. Or I should say, the expectation I have as an old school Unix administrator (that doesn't really care for Unix administration but can do it) and the current state of the art in Unix—or rather, Linux—administration.

Today's project was a consolidation of DNS—moving all the zone files to a central server and have that one serve the zones to the actual register name servers. That way, we have one location to make changes and from there they are propagaged out. Nothing too wild.

But it's not working. The zone transfers are failing. Big time. I'm reading the Cricket Book and I'm close to really mucking with the configuration files when this little voice in the back of my head goes Hey, check to see if there's a firewall running on the server. See, user expectation—I'm not used to Unix servers running firewall software as a default. I'm not running Windows for crying out loud.

And thankfully, it turns out to be a firewall issue. Otherwise I thought I was going crazy.

Tuesday, January 10, 2006


To whom it may concern:

It's Tuesday!

This is not Monday!


Repeat after me: TUESDAY!

If ever there's a day when both DNS servers fail, it's impossible to find a working serial port on a computer, my DSL connection fails, the A/C at The Office isn't functioning properly and the power cord falls out of my workstation for no good reason, it's MONDAY!


Please try to schedule any Hell Days for Monday.

Thank you.

What a fluke

From: Sean Conner <>
Subject: Re: The cable
Date: Sun, 8 Jan 2006 05:35:27 -0500 (EST)

It was thus said that the Great Bunny XXXXXXXX once stated:

Nice save on that cable. Very intuitive. Most of us would have looked for zebras. I recently changed out an outdoor light fixture (see attached), and when it didn't work, I almost didn't think to try a new lightbulb …

Helps to have a $5,000 network analyzer that can be used to test cables (even tell how long they are!).

Bunny and I (and yes, Bunny is her real name) have been exchanging emails about the network cable I recrimped. She seemed impressed that I would admit to making such a mistake, but hey, as mistakes go, it beats adding an external SCSI drive to your primary business computer without turning it off (a mistake an old boss of mine did).

[About 15 nanoseconds for the bits to schlep through this cable]

So I figured she might get a kick out of what a $5,000 network analyzer looks like. And I think it can do more than just measure the continuity and length of network cables, but darned if I've used it for more than that.

Wednesday, January 11, 2006

Cliff hanger

As if yesterday weren't bad enough, today I come home to find The Younger offering much sacrafice to the Great Porcelain God, due to stuff he may have ingested. Wlofie had called Poison Control and while what he ingested normally wouldn't be a problem, since he was making much offerings to the Porcelain God and complaining about abdominal pains, it was suggested we head off to The Emergency Room!

[It was an emergency!  I didn't have my camera on me, so I had to make due with what was available—which was Wlofie's phone]

We arrived just before 8:00 pm, with Spring flying to The Emergency Room from work. She arrived around 8:30, and shortly thereafter (after yet another offering to the Great Porcelain God) The Younger, accompanied by Spring, went into the bowels of the hospital.

At 1:30 am (yes, I am pre-dating this, as most of this happened Wednesday, and had there been wireless at The Emergency Room, I would have posted, but alas, there was none) Spring called, saying that The Younger's X-ray was inconclusive and next up was a CAT scan, and that the rest of us should probably head on back home as there was no telling how long The Younger and Spring would be there.

And that, alas, is all the information I have at this time.

So … how was your day?

Update on Thursday, January 12th, 2006 at 3:30 am

Doctors think it's a stomach flu, although why it took a CAT scan to figure that out, I don't know.

Thursday, January 12, 2006

Cliff hanger, Part II

Today was spent at The Hospital.

Not The Hospital whose Emergency Room I dropped The Younger off at. No. That Hospital does not have a pediatric section, so The Younger was transported to a different Hospital during the wee morning hours.

Now, since Spring has already mentioned one reason why he's there, I no longer have to dance about the issue of The Younger swallowing two kitchen magnets.


Yes, plural.

Now, as kitchen magnets go, these are tiny. Think of a push pin, one used to tack notices up to a bulletin board, but instead of a metal pin sticking out of it, it has a small round magnet to hold it to a refridgerator. We're talking something about 1/8″–1/4″ across.

And he only swallowed the magnet part, not the push pin part. Just the magnet part.

It was suggested by the doctors last night that The Younger might have pica which is a persistent craving for non-food like items.

Oh, and he also got a stomach flu, as if things weren't bad enough.

So anyway, today was spent at The Hospital.

The Younger was sent through X-ray enough times that he no longer needs a nightlight and the reason for the multiple X-rays throughout the day was to track the movement of the two magnets. The concern was if the two magnets attract each other in different parts of the intestines and lodge themselves against the intestine wall. Fortunately, the X-rays showed that not only where they progressing down his system, but they had already attracted to each other and travelling as one unit if I understood it correctly.

So, once the magnets are flushed from his system, The Younger can then return back home.

The Great Egress Incident

Since The Younger and Spring were transported to The Hospital from The Emergency Room via ambulance, Spring's car was still at The Emergency Room. I worked out a plan whereby Smirk would pick me up at The Hospital, drive me to The Emergency Room, whereby I would drive Spring's car back to the Hospital, then I would drive back home to get some sleep so I could go to work tomorrow.

Around 11:30 pm or so, Smirk calls me on my cell phone to say he's outside the main entrance to The Hospital. The main entrance to The Hospital is this large revolving door (it has motion detectors, and will start moving automatically). I nearly smack into the thing trying to walk outside.

It's not moving.

I try pushing it on the sign that says “Don't Push” but it won't budge. In fine print, the sign mentions something about the handicapped doors to either side of the revolving door. So I pick the one to the right, and try to go out.

It doesn't budge.

Nearby is a large flat disk with a stylized wheelchair and the words “Open Door” on it (or something to that effect—I don't recall exactly). It's made to be pressed, and I press it. I hear this light buzzing noise, only the door isn't opening. I try pushing the handicapped door, but it still doesn't budge.

Neither does the one to the left of the revolving door.

Through the windows, I can see Smirk's vehicle.

I try all three egresses again, pushing and shoving and otherwise trying to get to the other side of the doors.

Then I call Smirk on the cell phone, to inform him that I'm having some difficulty in leaving The Hospital. Through his laughter I'm told that he's seen my difficulty and suggests meeting him outside this Hospital's Emergency Room.

Friday, January 13, 2006

Monday, Part II

As if Tuesday wasn't bad enough (so was Wednesday and Thursday but that dealt with things outside of work), I arrive at The Office to one machine spewing out a sustained 20Mbps of traffic, and another one not responding at all.

For the one vomitting over the network (a customer's colocated box), I logged in, and saw the following errors:

eth0: Transmit error, Tx status register 82.
Probably a duplex mismatch.  See Documentation/networking/vortex.txt
  Flags; bus-master 1, dirty -1756485798(10) current -1756485798(10)
  Transmit list 00000000 vs. f7ed8480.
  0: @f7ed8200  length 80000042 status 00010042
  1: @f7ed8240  length 8000002e status 0001002e
  2: @f7ed8280  length 800005d6 status 000105d6
  3: @f7ed82c0  length 80000548 status 00010548
  4: @f7ed8300  length 80000036 status 00010036
  5: @f7ed8340  length 8000003e status 0001003e
  6: @f7ed8380  length 80000042 status 00010042
  7: @f7ed83c0  length 800005ea status 000105ea
  8: @f7ed8400  length 800005ea status 000105ea
  9: @f7ed8440  length 8000004e status 8001004e
  10: @f7ed8480  length 800005d6 status 000105d6
  11: @f7ed84c0  length 800005d6 status 000105d6
  12: @f7ed8500  length 800005d6 status 000105d6
  13: @f7ed8540  length 800005d6 status 000105d6
  14: @f7ed8580  length 800005d6 status 000105d6
  15: @f7ed85c0  length 80000042 status 00010042

Over and over again, for days—ever since it was rebooted last week. So I check Documentation/networking/vortex.txt:

Transmit error, Tx status register 82

This is a common error which is almost always caused by another host on the same network being in full-duplex mode, while this host is in half-duplex mode. You need to find that other host and make it run in half-duplex mode or fix this host to run in full-duplex mode.

As a last resort, you can force the 3c59x driver into full-duplex mode with

options 3c59x full_duplex=1

but this has to be viewed as a workaround for broken network gear and should only really be used for equipment which cannot autonegotiate.

I know Dan the network engineer prefers not to have auto negotiation as he's seen problems with it, so I call him up, and ask him to set the switch port the machine is plugged into as 100Mbps auto-negotiate. Once he did that:

eth0: Setting full-duplex based on MII #24 link partner capability of 4101.

And the traffic immediately cleared up (I suspect that when the box was first plugged in, the switch port was set to auto-negotiate, but some time afterwards, Dan set the switch port not to negotiate, but since the negotiation already happened, no errors were reported. It was only after the box was rebooted last week that the problem cropped up).

The second box was also a colocated machine, and the customer there just told me to power cycle it. Instead of just powercycling the machine, I hooked the crash cart up to it, just to see if any error messages were on the screen.

Only the screen was blank.

Couldn't get it to unblank at all.

The machine must be hosed, so I power cycled it.

And that's when I saw the other video interface.


The rest of the day was pretty much like that—just as I clear one problem, another would pop up. All day I was busy with problems.

It was like Monday Tuesday all over again …

More updates on the tarpit

Labrea is actually logging about half a gig a day. Over a 24 hour period (from about 6 am Thursday to 6 am today) I'm tarpitting 82,359 connections across 2,059 unique IP addresses (24,252 connections from a single IP address). And while the number of network ports being accessed has increased a bit, it's the Microsoft specific ports that are still the most popular targets (with 72% of the scans):

Top 10 ports captured by Labrea in the past 24 hours
Port # Port description # connections
Port # Port description # connections
139 NetBIOS Session Service 24941
445 Microsoft-DS Service 23013
1433 Microsoft SQL Server 6772
4899 Remote Administration 5620
135 Microsoft-RPC service 4722
80 Hypertext Transfer Protocol 3697
8080 Hypertext Transfer Protocol—typical alternative port 1686
7212 (unknown) 1683
8000 (unknown) 1471
10000 (some web based control panels use this port) 951

The program I'm using to generate the stats is written in Perl, and it took about 4 hours to run over a day's worth of data (the machine that does the tarpitting isn't the fastest machine we have, but it's more than enough to dedicate to just running LaBrea). I definitely want to write a program to process LaBrea data in real time.

Saturday, January 14, 2006

The passing of stuff

I'm at the hospital, which unfortunately does not have Internet access (which sucks, and it's obvious that I'm posting this some days after I wrote this).

Good news for The Younger: he seems to have gotten over his stomach flu.

Bad news for The Younger: the magnets have been stuck somewhere in the large intestines for the past 48 hours. And he won't be released until the magnets are released.


Processing realtime data from LaBrea

Since I'm stuck at The Hospital, and The Hospital doesn't have wireless, I might as well work on the real-time LaBrea data processing program. Loaded up the laptop with program fragments and libraries I may need and brought along my copy of Advanced Programming in the Unix® Environment by W. Richard Stevens, and am coding up a storm.

And yes, I'm coding this up in C. Not like I'm going anywhere in the next couple of days.

LaBrea spits out lines like:

1136832897 Initial Connect - tarpitting: 3334 -> XXX.XXX.XXX.XXX *
1136832897 Initial Connect - tarpitting: 3339 -> XXX.XXX.XXX.XXX 139
1136832897 Persist Trapping: 3334 -> XXX.XXX.XXX.XXX 445 *
1136832898 Persist Activity: 45285 -> XXX.XXX.XXX.XXX 135
1136832898 Persist Activity: 34589 -> XXX.XXX.XXX.XXX 135 *
1136832898 Persist Activity: 3862 -> XXX.XXX.XXX.XXX 135
1136832898 Persist Activity: 4459 -> XXX.XXX.XXX.XXX 139 *

There's some other stuff that's logged, but I'm primarily looking for lines like the above. Which means the information I can store will look something like:

struct tprecord
  IP     src;
  Port   sport;
  IP     dest;
  Port   dport;
  time_t start; /* first time we see src:sport-dest:dport */
  time_t last;  /* time of last activity of src:sport-dest:dport */
  size_t packets;

Now, since we'll be handling a lot of connections, we need to store these structures in a way that is easy to search through. Could use a hash table, and we are keying off the src:sport→dest:dport tuple, and that's 10 bytes of pretty much random binary data, which is good for this type of thing. But it's an open ended hash, not knowing how many entries we'll be handling. There are several methods to handle overflows in a hash table, but that still means at some point doing a linear search, and then there's the overhead of having a hash table.

There's also storing the data in a tree. In fact, I even have code in C to manage a balanced tree (taken from Knuth). But the downside is—I only have code to add to the tree, not to delete from it. And it was hard enough to write as is.

And there's still the overhead of maintaining a tree structure.

So I'm using arrays. Less overhead and straightforward to manage. The only trick is keeping the array sorted in some order, but as long as you compare the structures in a consistent manner, that's not really an issue (and by the way, I'm sorting by source address, port, destination address and port, in that order).

To make it even easier, I have an array of structures (see above), and an array of pointers to said structures, and it's the pointer based array that is sorted (since it's faster to swap pointers than swap whole structures). And once sorted, you can use a binary search to find the record.

The default binary search in C, bsearch(), returns either a pointer to the found record, or NULL if not found. Good for most uses, but not quite what I want. In addition to either finding the record or not, I'd also like to know where in the array the record was found, and if not, where it would appear if it was in the array. And C's bsearch() does not return the index.

Why do I want that?

My thinking is that if the search (reguardless of what type of search it is) didn't find the record in a sorted array, it has the information as to where the missing record should be. And with the information as to where, it would be “trivial” to add the record at the right point in the array.

And that beats adding the new record at the end, and restorting (C's qsort() is Quick Sort, which on average is O(n lg n) running time, but the worse case is O(n2) and the worst case is then the array is mostly sorted to begin with, which is why I'd rather not resort after each addition).

And try as I might, I could not get a binary search to work that would also return the index. For now, I replaced the search with a linear search, until I can comb through some reference materials at home.

The other problem I'm having is the array manipulation code doesn't quite work. When the program starts, it has space to store X records. If it runs out of space, it increases the size of the various arrays until it hits some large number of records, then it will go through, purge records fitting some criteria.

Only it's not working properly.

Got a lot of work done on the program though, and it was nice to get into The Zone (even with a headache). At this point, I'm headed home for sleep and what looks to be yet another day at The Hospital.

Sunday, January 15, 2006

And this too, will pass … the only question is when!

Back at The Hospital. The Younger is over definitely over his stomach flu, but the magnets are still stuck.

Normally on Sundays I run a D&D game for the family and The Kids were looking forward to playing this week, but frankly, I don't want to run a session at The Hospital. For one thing, I have a pile of reference material I use during a session and there isn't a good enough surface for me to use. And the smoke breaks (for both Spring and Wlofie smoke) will be longer (since they have to head outside The Hospital instead of just outside Casa New Jersey), leading to less actual gaming time.

Second, I'm still in a programming head space and don't want to loose being in The Zone.

Third, Spring fell asleep as soon as I got there, and Wlofie was home sleeping, and that's half the party. I'd rather not run a session with that many missing party members.

In an effort to get the magnets moving, The Younger is being forced to eat (not only by us, but through The Hospital staff) even though he complains of stomach pains.

I have this feeling that it's going to be a long hospital stay.

Some notes on a binary search implementation

You would not believe how hard it was to write a binary search that returned the correct index for a missing record in an array.

Even with Programming Pearls by Jon Bentley (reference book of the day).

Binary search is a deceptively simple algorithm that is easy to get wrong (be especially careful when searching empty arrays). And of the binary search implementations I've seen, when they do return the index in the array, it's to an element that's actually there, and some other value (like 0 or -1) if the element isn't there. I've yet to see an implementation that returns an index reguardless (plus an indication if the element was found or not).

So I spent several hours getting a binary search to return an index even if the element wasn't found. Before looking at my solution you should at least try coding it (I should note that the code is pulled from the real-time LaBrea data processing program I'm writing, which just enough changes to get a single file to compile and with as little change to the source code as possible, so some of the names may not make that much sense, but the logic should be clear).

Squishing bugs

I've been simplifying the real-time LaBrea data processing program. One simplification: don't try to grow the arrays. If I'm trying to grow the arrays and I'm going to stop at some point anyway, why not just allocate as much as I'll ever use when the program starts? It gives you a definite upper bound (and one that can be set from the command line) and boy, does it simplify the code.

I also got the binary search routine working correctly after a few hours.

And I had to remind myself just how C's qsort() works, and make sure I know that (in my program at least) I'm getting pointers to pointers to structures, and not just pointers to structures (when I fixed that, I started getting correctly sorted arrays).

The only outstanding bug I have right now deals with adding a record after the array has been purged of old records. The problem is that I'm using an index from before the purge. I should be able to get this running on the LaBrea tarpit system in the next day or two.

Monday, January 16, 2006

Sexism is alive and well even today

The Younger has yet to pass the magnets, so he's still in The Hospital.


And so far, despite repeated corrections, most of The Hospital staff still refer to The Kids as “girls.” Okay, understandable given the shoulder length hair both have, and slender bodies.

But repeatedly?

“So, is she with you?”

“Yes, he is.”

“Should I give her a badge?”

“Yes, you can give him a badge.”

“Here you go little girl.”


(Not to mention that one staff member not only though The Younger was a girl, but still wore diapers! I mean, yes, we're waiting for an event from down there, but The Younger can hit the bathroom for that. Sheesh!)

Didn't we learn that long hair isn't an indication of gender during the 70s?

It's enough to make me want to put my head in my hands and sob uncontrollably.

Stupidity is alive and well today too!

On the way to work (after dropping off Wlofie and The Older at The Hospital), I stopped off at a gas station to fill up Lake Lumina. There are two pumps next to each other, both free. I'm trying to negotiate my way there when an SUV cuts in. But does the soccer mom driving said SUV pull all the way forward?

No, of course not.

She pulls up to the first pump, and stops.

Would it hurt her to pull up to the next pump, free of any vehicular obstruction and fill up there?

Yes, of course it would.

So I have to drive around this bree munching Yuppie soccer mom and circle around to some other free pump.

Then, when said bree munching Yuppie soccer mom finished filling up her SUV (I didn't actually see her get out of the vehicle and pump any gas, although I did see her talking on a cell phone), does she pull forward to circle around the lot to leave?

No, of course not.

Instead, she backs up and tries to negotiate a three point turn right there in front of the pump.

It's at this point that I want to put her head in my hands and shake violently.

Monday, only this time, on a Monday

From: Dan the network engineer <XXXXXXXXXXXXXXXXXX>
To: Sean Conner <>
Date: Mon, 16 Jan 2006 11:12:05 -0500

Hey Sean,

XXXXXXXXX's traffic issues have not been solved, and, in fact, are still causing some major issues.

I believe this server has been hacked and needs to be taken off-line …

Okay, see! This is a Monday thang …


Not that bad, as these things go

Well, the server was hacked, but it looks to be a customer account was compromised, since the executables where owned by a customer account, the processes were running on unpriviledged ports, and the server was being used as part of denial of service attacks, with executables hidden under a hidden directory in /var/tmp.

Fortunately, the system hacked is running Linux without module support, so patching system calls to hide activity is impossible without a reboot (which would be noticed).

And as always, it could have been worse.

Tuesday, January 17, 2006

“It comes in two flavors, ‘icky’ and ‘yucky.’”

The magnets are still stuck in The Younger. And this morning he was in pain. Quite a bit of it too. Hopefully the pain means they are on the move, but we won't know until later today after his insides are scanned once again.

If the magnets haven't moved, tomorrow the doctors go in.

Here's hoping the magnets leave his system on their own.

Real-time LaBrea data processing program

I finished writing the real-time LaBrea data processing program (most of it last night at The Hospital) and the final few bugs were real doozies.

The program works by getting data from LaBrea, then it looks up the connection and updates the information accordingly. The code pretty much looks like:

void start_tarpit(time_t stamp,char *line)
  struct tprecord *exists;
  struct tprecord  rec; 
  size_t           index;   

  exists = tpr_search(&rec,&index);
  if (exists == NULL)
    exists = pull_free_record();

  exists->src = rec.src;
  exists->sport =; 
  /* ... */



struct tprecord *pull_free_record(void)
  if (g_poolnum == g_poolmax)


void record_add(size_t index,struct tprecord *rec)
  if (g_recnum == g_recmax)
        &g_rec[index + 1],
        (g_recnum - index + 1) * sizeof(struct tprecord *)
  g_rec[index] = rec;

tpr_search() is the binary search routine I wrote the other day, and as you can see, it returns the index to where the record is in the array, or to where it should be. pull_free_record() just returns the next free slot in the structure array, and if there are no slots available, it does a removes some older records according to some criteria. And record_add will add the record to the pointer array, also removing older records if there is no space left.

Some records are deleted. All remaining records move about. The pointer array is resorted. So between the time

exists = tpr_search(&rec,&index);



index may not be a valid index anymore!


(Never mind the fact that one of the two calls to do_forced_garbage_collection() is redundant)

Simple enough to fix once I knew what was going on.

Another bug dealt with named pipes. Instead of directly piping the data from LaBrea to ltpstat (what I call the read-time LaBrea data processing program), I decided to go through a named pipe, which would allow me to start and stop either one independantly from the other.

Now, I'm testing my program, running cat sample > labrea.pipe in one window, and ltpstat --input labrea.pipe in another. It's working, until the data in sample runs out and cat closes its side of the named pipe.

Now, the code that reads in the data is in a library I wrote, and it just assumes when a read() returns 0, that it's the end of the file, and marks it as being closed. ltpstat ignores the “end of file” status, and keeps trying to read a now-closed file. We get into a busy loop and the system load shoots up. Also, if I now try to pump more data through the named pipe, ltpstat ignores the data.

Even when I modify the library code to not mark “end of file” when there's nothing to read does nothing, as from that point on, read() just returns nothing anyway and seems to be a “feature” of Linux (or Unix—I didn't have Advanced Programming in the Unix® Environment with me to look this up), so I restructure the main loop:

  in = openinput();

That was all fine and good, until I threw signals into the mix.

I use signals to tell ltpstat (what I call the read-time LaBrea data processing program) to dump various information—SIGHUP to print the number of connections and unique IPs being tracked, SIGUSR1 to do a raw dump of all the data accumulated so far and SIGUSR2 to generate a more or less human readable dump.

But signals are basically interrupts. And operating system theory states that one process should never find another process with its pants down (so to speak—and I should warn you—the paper I linked to is way more technical than I've gotten here). Signals and system calls interact (that is, if a process is signaled while making a system call into the kernel) in one of two ways—the system call will simply fail, or it will be restarted automatically. And one can select which method to use.

If I elected to have the system call fail, and ltpstat would fail, either to open (a system call) the named pipe, or in reading (another system call) the named pipe after I signal the program.

If I elected to have the system call restarted, and the signal handlers I set up would never get called (due to the way I handle the signals).

I ended up reimplementing two routines from my library (which is used in more than just this program) for just this program. I select for the “system call fail” method, check to see if the system called failed due to a signal, and if so, check for the signals, and try again.

Again, this took a few hours to track down.

But now the program works, and I can finally get real time statistics from LaBrea.

Last minute update!

I just received a call from Spring—the mangets are in The Younger's stomach! (why this wasn't found out sooner, I don't know). So, because it's in his stomach, he's being moved yet again to another hospital, one that has the equipment that might get the magnets from his stomach.

I'll be heading there once I get word which hospital it is.

Some initial data from a real-time LaBrea data processing program

While I'm waiting a call back, some more on LaBrea.

Yesterday (from January 16 at 06:28:25 to January 17 08:54:50) LaBrea generated 1.1G of log data, and it took full five minutes to run grep 'Initial Connect' daemon.log.0 | wc -l (255,344 new tarpitting connections by the way).

LaBrea was also running at full speed, maxed out at 64Kbps bandwidth to keep all these connections tarpitted (the maximum I set LaBrea to use, by the way).

[NETWORK TARPIT bandwidth usage]

That first large dip in the graph (the one around 6:30 in the morning) is probably due to the system attempting to rotate a 1.1G log file. The second dip, at the right (around 3:00 pm) is when I restarted LaBrea so its logging information would go through ltpstat. After an hour of running:

Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 15:55:59 2006 Running time: 1h
Pool-max: 1048576
Pool-num: 24322
Rec-max:  1048576
Rec-num:  24322
UIP-max:  1048576
UIP-num:  1282
Reported-bandwidth: 40 (Kb/sec)

And after two hours:

Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 16:56:19 2006 Running time: 2h 20s
Pool-max: 1048576
Pool-num: 33326
Rec-max:  1048576
Rec-num:  33326
UIP-max:  1048576
UIP-num:  1632
Reported-bandwidth: 40 (Kb/sec)

And right this second:

Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 18:37:19 2006 Running time: 3h 41m 20s
Pool-max: 1048576
Pool-num: 42931
Rec-max:  1048576
Rec-num:  42931
UIP-max:  1048576
UIP-num:  2148
Reported-bandwidth: 40 (Kb/sec)

Okay, pool-max and rec-max are the maximum sizes for the structure array and pointer array, and both should always be equal at all times (I'm displaying this number more for debugging purposes than anything else), while pool-num and rec-num (which should also be equal at all times) represent the current number of connectioned tarpitted. I also keep track of unique IPs, which just now is currently 2,148 (out of 1,048,576 that I can store). I also just found out that IP address has 4,809 connections currently tarpitted (and in the few seconds it took to do that query, five more connections were tarpitted).

I'll be releasing this code in the next few days, when I can write up some documentation and slap on a license.

This just in …

Just now received a call from Springthe trip to the other hospital has been postponed until tomorrow morning.

That is all.

Wednesday, January 18, 2006

No news is good news, I suppose

The Younger is waiting to be transported to the next hospital in line. The magnets are still in his stomach waiting to be removed.

So, hurry up and wait …

LaBrea ad-naseum

Just because I was curious, I ran the Perl script that parses the LaBrea logs on the 1.1G logfile from the other day. I knew it would take at least four hours or so to run.

I then created a modifed version of ltpstat to parse this file (it's the output from syslogd, hense some slight parsing modifications were required) and ran it at the same time.

What I found interesting (during the actual run of both programs) is that my program had a larger virtual memory footprint over the Perl version (easily seven to eight times larger) but the resident set size (the amount of physical memory being used—the rest not physically allocated or shoved off into swap space) of my program was half that of the Perl script. In retrospect this was expected—Perl was growing the data as it was being generated whereas my program allocates the whole thing at once, but my program has less overhead in keeping track of said data.

And ltpstat is faster than the Perl script even if it isn't gathering the stats in real-time— 3 hours, 22 minutes and 24 seconds to run vs. 6 hours, 39 minutes 49 seconds—almost half the time. I didn't see how much memory the Perl script was using just prior to finishing, but I can't see how it would be less than ltpstat.

The instance of ltpstat I started yesterday is still running:

Start: Tue Jan 17 14:55:59 2006 End: Wed Jan 18 14:55:59 2006 Running time: 1d
Pool-max: 1048576
Pool-num: 388929
Rec-max:  1048576
Rec-num:  388929
UIP-max:  1048576
UIP-num:  20298
Reported-bandwidth: 64 (Kb/sec)

Looks like I may break a million connections sooner than I expected.

Thursday, January 19, 2006

This is the stay that never ends

Latest update on The Saga of the Younger: he's been moved to the next hospital in line, and there the doctors discovered that the magnets are stuck in an ulcer in his stomach, and the only way to get them out is surgery. We have no time frame for when said surgery will happen though.

Updates as I get them.

Friday, January 20, 2006


The Younger is out of surgery. That is all I know at this time.

Update a few minutes later …

The magnets did get separated—one ended up in his stomach and the other one ended up in his intestines. They did attract each other and the attraction was enough to cause them to borrow through tissue.

They've been removed (and were quite corroded from what I understand) and the holes sewn up. He's in recovery for the forseeable future. So now it's just waiting around until he heals up.

Saturday, January 21, 2006

Quick game of ketchup

I have internet access for the evening, courtesy of having to come to work. Trying to catch up on the most essential emails and such, in between working cases.

I love you people. Thank you so so so much for the messages of concern, wishes, prayers, and other good vibes, as well as information on polar bear blubber and vitamin poisoning. Thanks also for offers of assistance. We've got things covered, but truly appreciate it.

Magnet news:

On the 19th the team at West Boca sent an endoscope down his gullet, under general anaesthesia. They found an ulcer with a shadowy something in the middle that was likely the magnets. That something in the middle couldn't be gotten out. Surgery required.

The surgery occurred yesterday, the 20th, and took longer than anticipated—it was about 2 hours. rhonan had it right—one magnet had gone down the small intestine, and the other had stayed in the stomach, and they attracted. The ensuing irritation and vomiting episode caused them to begin tunneling toward each other, scar tissue forming a tube. Surgeons had to disconnect the tube, remove the magnets, and fix the holes. They were thrilled at how fixable it was, though - no ancillary damage like necrosis or sepsis or any of that. It could have been far worse.

Afterward, straight into ICU and morphine. It's not fully effective against the pain, though. He got out of ICU today and is still convalescing, no food or liquid yet. The innards have to heal first.

My weekend is starting after tonight's shift, and I'll most likely be spending all of it at the hospital. I'll certainly be there after shift tonight. Will continue to post phone-to-email-to-LJ updates as necessary and able.

Spring Dew: A quick game of ketchup

Not much more to add to this (except I don't have a phone-to-email gateway to update this site).

Are you tired of network tarpits yet?

You would not believe how hard it was to write a binary search that returned the correct index for a missing record in an array.

Some notes on a binary search implementation

A week later, and I finally have it working.

One technique used to debug a program is to have another program that does the same thing, but implemented using a different method or language or both. And I did. I ran the Perl program I had over the 1.1G log file, then ran ltpstat over the same log file and got two different results.

Not good.

ltpstat returned 2% more connections than the Perl script. Getting a dump from the currently running version on the LaBrea system and cleaning the output showed a 2% difference again.

So I spent the past week trying to track down the problem. It was obvious that ltpstat was storing duplicate records, but why was a different matter. My testing sample of about 1,100 connections is apparently too small to completely test the program, so I had to test using the 1.1G log file which has approximately 230,000 connections.

To help debug this problem, I wrote a linear search and would call it as well as the binary search. If both agreed, then I would return the information, otherwise, I would log the discrepency, do the search again, then exit. The reason for doing the search a second time? So I could set a breakpoint there, and let the program run for a couple of hours until it triggered. Then I could step through both searches to see where the problem was.

Yup, each run took several hours to trigger the bug.

I ended up testing four different binary search routines (including the original one I thought worked, plus one I modified from The Standard C Library, plus two other versions I wrote) before sitting down and working through things on paper.

And I still missed corner cases.

But finally, I tested my final version it against the Perl script and only had 122 discrepencies out of some 230,000 records (or 5% of 5%—too small for me to worry about after spending a week on this).

I took a snapshot of the currently running version (which had been running for a bit over three days now), cleansed the output of duplicates, and the final tally was 416,230 connections from 12,911 unique IPs. Again, nothing surprising about the ports being attacked:

Top 10 ports captured by Labrea in the past 3 days
Port # Port description # connections
Port # Port description # connections
139 NetBIOS Session Service 160,799
135 Microsoft-RPC service 108,958
445 Microsoft-DS Service 67,506
80 Hypertext Transfer Protocol 23,921
4899 Remote Administration 9,225
22 Secure Shell Login 7,253
1433 Microsoft SQL Server 6,503
8080 Hypertext Transfer Protocol—typical alternative port 3,717
3128 Squid HTTP Proxy 3,329
1080 W32.Mydoom.F@mm worm 3,150

And again, the Microsoft specific ports account for 81% of the scans. I'll need to discuss with Smirk about blocking those ports in the core router. If nothing else, LaBrea is giving me an indication of which ports to block.

Sunday, January 22, 2006

For once this year, a post that has nothing to do with tarpits or hospitals, except for this self-referential title mentioning that this post has nothing to do with tarpits or hospitals.

It's weird. I have my email address plastered all over my website here, and yet, I'm receiving a bazillion spams for a type of sexual aid at my private email address. But at my oh so public email address?

Nothing what so ever about this particular sexual aid.

How odd.

Monday, January 23, 2006

“Waaaaah waaaaah waaaaah”

I remember having this weird dream about ten years ago or so. In this dream, my boss (it wasn't you Smirk) walked up to me, opened his mouth and said “Waaaaah waaaaah waaaaah waaaaah waaaaah.”

“What?” I said.

“Waaaaah waaaaah waaaaah waaaaah,” said my boss. He then went around the office continuously saying “waaaaah waaaah waaaah,” even if he wasn't speaking to anyone in particular.

I do recall in my dream that a bunch of us were trying various methods to shut him up, since we all found his “waaaaah waaaah waaaaah” very annoying. After what seemed like hours, it dawned on me that this wasn't the normal behavior of our boss and that something else might be to blame. It was then I realized I was dreaming, and woke up. To my alarm clock, which was going “waaaah waaaaah waaaah waaaaah waaaaah” and had been for the past 40 minutes.

Yes, I'm a heavy sleeper.

I also found out (about the same number of years ago) that my alarm clock would simply give up after three hours of me hitting the snooze button (not only did I hit the snooze button for three hours, the alarm clock itself was on the other side of the room—yes, for three hours, every nine minutes, I would roll out of bed, take two steps across the room, hit the snooze button, two steps back and roll back into bed, without disturbing my sleep!)

I no longer use that alarm clock—oh, it still works, it's just that the hour button has broken off (personally, I would have expected the minute button to break off) so it's rather difficult to set. I've been using my cell phone as an alarm clock. No snooze mode, so I have to be semi-conscience to manually set the alarm, but the alarm itself only sounds for a minute, which may be too short a duration.

I only mention this to explain to Smirk was I was nearly three hours late to work today.

Tuesday, January 24, 2006

Update for those playing at home

The Younger started in on liquids today, and it looks like he'll be eating solid food starting tomorrow. That, and he's flirting with the nurses, so he's healing right up. I'm expecting he'll be in The Hospital for the next few days or so.

A thought process …

I have to wonder the thought process of a script kiddie that runs the SSH22 Scanner (which tries to log into servers via ssh using a series of precanned userids and passwords) as fast as it can. Doing so only brings attention to the fact that it's sucking up network bandwidth and loading down the box, gaining the attention of the sysadmin, thus negating the purpose of running such a program in the first place.

I also have to wonder at the thought process of a script kiddie that logs back into a box after loosing the connection and not bothering to see if anyone is logged in as root before starting the SSH22 Scanner back up.

Wednesday, January 25, 2006

So when does 2 not equal 2?

If someone was to drop a bomb on this building, it would wipe out 50 percent of the Lisp community. That would probably be a good thing. It would allow Lisp to start over.

That wasn't exactly what John said, but it was pretty close. Again, the main thing here was somebody with impeccable Lisp credentials saying that Lisp was far from perfect and in fact needed to do better than Common Lisp. The fact that Patrick, Henry, and John all argued essentially the same thing was a wake-up call, I think.

Finding Lisp: ILC 2005, Wednesday report (late)

Now, I have this love-hate relationship with Lisp. I love the idea of Lisp, and it is a cool language, but I hate the implementations of it (it doesn't play well with other languages), lack of syntax (which expresses itself in Lisp as a metric buttload of parenthesises), the nasty low level details one has to keep in mind (really! In Common Lisp, 2 (this particular instance of the number) is not necessarily equal to 2 (this other instance of the number)—I kid you not!) and the rather archaic way it still operates (CAR and CDR are remnents from the very first implementation where one took the “contents of the address register” for the current element in a list, and “contents of the decrement register” for the rest of the list—even though you now have FIRST and REST LAST (oh, it's called LAST even though it returns the rest of the list) REST (apparently, I had it right the first time, and LAST is used for something else in Lisp—sigh) nobody ever uses those).

Oh, and to get any real speed out of Lisp you have to forego using lists (which is what Lisp stands for—“LISt Processing”) and use more specific data structures (never mind the fact that Lisp is still taught as if it only has lists).

The “John” mentioned above is John McCarthy, inventor of Lisp, which is one of the two oldest programming langauges (the other being FORTRAN), and if he, of all people, doesn't have much positive to say about Lisp, then what chance does it have of gaining wide spread acceptance in the industry (Greenspun's Tenth Rule notwithstanding)?

Thursday, January 26, 2006

He's Out!

The Younger has left the hospital!

Thank God that's over.

And if The Younger ever eats another magnet again, I'll kill him! Sheesh!

(/= 2 2)

My recent post on Lisp generated some heat. First, from Zach:

To: Sean Conner <>
Subject: (/= 2 2)
Date: Thu, 26 Jan 2006 10:56:38 -0500

I came across your Lisp post today, and I wanted to comment on a couple things.

FIRST and LAST are not analagous to CAR and CDR; LAST returns some part of the tail of a list (depending on its optional second argument). The list-oriented analogue of CDR is REST. (You had that, but crossed it out.)

As for 2 not necessarily being equal to 2, there are many equality predicates in Common Lisp. For example, comparing 2 and 2 with the EQL, EQUAL, EQUALP, and = functions will all return true (or there is a bug in the implementation). It is only the object “same, identical” predicate EQ that will sometimes fail on two apparently identical numbers, and even then it's only likely when the number enters the bignum range. Fortunately, most documentation and tutorials give the good advice to use EQL (not EQ) as a default “sameness” test. A novice would have to take a fairly peculiar learning path to get burned by using EQ on numeric values.

We had a bit of an exchange and I ended up writing:

From: Sean Conner <>
Subject: Re: (/= 2 2)
Date: Thu, 26 Jan 2006 15:18:07 -0500 (EST)


Odd thought just cross my mind. It may not be fully applicable to Lisp and it's half formed, but I'll try to get it out anyway.

Lisp is a dynamically typed language, and a lot of programmers like dynamic typing since it frees them from the burden of having to declare the type of a variable [1] (often times the excuse is “I'm prototyping—I don't know what types I'll end up using.” [2]). Having to declare the type of a variable is seen as pointless pedantry and something the compiler can keep track of [3].

I feel the same way about the equality functions in Lisp—pointless pedantry and something the compiler should keep track of.

  1. Yes, in Lisp, values have types, not variables.
  2. A weak excuse in my opinion. Strikes me as being muddle headed about what the program is supposed to do. I should mention that I actually liked Ada, and tend to prefer strict type checking, if only to make sure I don't make bone headed mistakes.
  3. Fair enough argument, but I'd prefer to state my intent to the compiler anyway. The more information it has, the better off it is.

Just how much of a pointless pedantry is Lisp's equality functions? Well, there are five defined in Common Lisp (quoted from Paul Graham's ANSI Common Lisp):

(eq object1 object2)
Returns true iff object1 and object2 are identical. [It doesn't say what it means by “identical” but given that (eq 2 2) can be false in a conforming implementation of Common Lisp, I'm guessing “identical” means “same address”]
(eql object1 object2)
Returns true iff object1 and object2 are eq, or the same character, or numbers that would look the same when printed. [So even here, (eql 2 2.0) is still false, at least, according to Rodney A. Brooks Programming in Common Lisp.]
(equal object1 object2)
Returns true iff object1 and object2 are eql; or are conses [one element of a list] whose cars [first element] and cdrs [rest of the list] are equal; or are strings or bit-vectors of the same length (observing fill pointers) whose elements are eql; or are pathnames whose components are equivalent. May not terminate for circular arguments. [So, I guess this means that (equal 2 2.0) may be false as well]
(equalp object1 object2)
Returns true iff object1 and object2 are equal, char-equal[um … I guess there are more than just the five equality functions then], or =; or are conses whose cars and cdrs are equalp; or are arrays with the same dimentions whose active elements are equalp; or are structures of the same type whose elements are equalp; or are hash tables with the same test function and number of entries whose keys (as determined by the test function) are all associated with equalp values. Reasonable to assume that it may not terminate for circular arguments. [So, if you want to compare arrays, you can't use equal, and still(equalq 2 2.0) is false.]
(= n1 &rest ns)
Returns true iff the difference between each pair of arguments is zero. [Finally! We have an equality test where (= 2 2.0)may be true!]

Why so many? I don't even want to hazzard a guess (perhaps Lisp programmers like micromanaging equality whereas C programmers like micromanaging variable type information—oh darn, I hazzared a guess).

My friend Andrew wrote in:

To: Sean Conner <>
Subject: An apology for Lisp
Date: Thu, 26 Jan 2006 14:58:44 -0500 (EST)

If you still think Lisp is wrong, then consider this: is 2.0 equal to 2? Why or why not? I think reasonable people could disagree, and the designers of Common Lisp did too; that's why there are multiple equality predicates with different semantics.

As for avoiding lists to get “real speed”—you say that like nobody uses O(N) complexity data structures to do real work in C! Lisp lists are what they are; for many applications they're fast enough. Otherwise, Common Lisp has arrays, hash tables, and more.

Okay, but if there are languages out there that can do type inferences on data, then certainly languages can also do equality inferences with data as well. Or perhaps I've been corrupted with Perl and it's very lax approach to equality …

And to Lisp's other data structures? Most introductions to Lisp I'm familiar with start with lists, and rarely, if ever, mention any past that. For instance, Rodney Brooks' Programming in Common Lisp, a 12 chapter book and arrays are first mentioned in Chapter 12! (“Other Features of Common Lisp”). And even though Paul Graham mentions arrays, trees and hash tables in Chapter 4 of his ANSI Common Lisp, the chapter title doesn't do them any service in my opinion (“Specialized Data Structures”—I really didn't know arrays were a specialized data strcture). Also, the way lists are described ad-naseum in Lisp books, which special pains to point out the structure of the “cons cells” that it seems like Lisp is still stuck with a 60s implementation.

Oh, it's still useful, just not as useful as I expected

By using ltpstat I've been able to see that the LaBrea tarpit isn't quite as effective as I first thought. Yes, it does slow down scans, but not quite as much as one thinks. I'm guessing that scanning software now includes the “timeout” concept—if a connection takes too long, drop the connection and move on.

A few days ago I added a feature to ltpstat to remove entries that have not seen any activity for over an hour (default setting). After running the tarpit for over a day, I see the following stats:

Jan 27 01:57:08 ltp ltp-report: Start: Wed Jan 25 17:27:55 2006 End: Fri Jan 27 01:57:08 2006 Running time:  1d 8h 29m 13s
Jan 27 01:57:08 ltp ltp-report: Pool-max: 1048576
Jan 27 01:57:08 ltp ltp-report: Pool-num: 107287
Jan 27 01:57:08 ltp ltp-report: Rec-max:  1048576
Jan 27 01:57:08 ltp ltp-report: Rec-num:  107287
Jan 27 01:57:08 ltp ltp-report: UIP-max:  1048576
Jan 27 01:57:08 ltp ltp-report: UIP-num:  2558
Jan 27 01:57:08 ltp ltp-report: Reported-bandwidth: 32 (Kb/sec)

Okay, I've “captured” 107,287 connections. But how many of those are still active?

Jan 27 01:58:32 ltp ltp-report: Removing records with no activity for the past  1h
Jan 27 01:58:32 ltp ltp-report: ... keeping 11180 records with activity since Fri Jan 27 00:58:31 2006

Well then. Over 96,000 connections were no longer “active” and of the 2,558 machines doing the scanning, some 2,200 had moved on.

So it looks like the LaBrea tar pit is really only useful to see what's being attacked, and which machines on the Internet are really doing the attacking (so far, seems to be quite tenacious in scanning).

And the ports being scanned? Again, it's the Microsoft specific ports as usual. No use making a chart this time.

Friday, January 27, 2006

Rabbit Hole Day

This article is intended to present a set of basic rules for writers to follow when creating “Conspiracy” or “Fringe Theory” texts. I hope that no-one is tempted to use them for “real world” applications—except for spotting the faults in existing material. However, real “fringe theory” buffs don't need me to teach them how to think. (But I've done so anyway, using my personal orbital mind control laser.)


Today is The Second Annual LiveJournal Rabbit Hole Day—a day where LiveJournal entries were supposed to be fun, wierd, unexpected, different, in honor of Charles Lutwidge Dodgson's 174th birthday (although he's not around to enjoy it). I thought it sounded like a neat idea, so I thought I'd try it here, even though this isn't LiveJournal.

Only you can see that I didn't quite get a round toit (those seem to be hard to find lately) and that's due to not having an idea of what to write about until too late, and then running out of time.

An idea I was trying to come up dealt with one Wolfgang Amadeus Mozart, seeing how it's his 250th birthday (and he too, is not around to enjoy it), but I had no good ideas. Nothing jelled until I found out that it was also the birthday of Captain Edward John Smith, R.N.R. (who too, isn't around to enjoy the celebrations, mainly because he was obligated to go down with his ship). Once I saw that, then it hit me—a conspiracy theory tying all three together.

I didn't get a chance to work on this at work, so I thought I might be able to at the weekly D&D game (since I rarely pay attention to the game anyway, and Bob, the DM is fine with that) but this week, of all weeks, his computer took a dive and it took several hours to help him debug the issue (his copy of Panda Network Antivirus updated itself and the update hosed his computer hard).

So there went that idea.

So no Rabbit Hole Day for me this year.

Maybe next year.

Saturday, January 28, 2006

Tunnelling through the Internet

Some of the networking we're doing at The Office is beginning to scare me, and not because it's complex, but because … well … it's complex and it seems like a kludge.

Basically, we're providing a backup Internet connection for one of our customers via DSL; they have an existing Internet connection with another provider. We are also giving them a block of IP addresses to use even when their primary Internet connection is working! So we need to route a portion of our network through some other provider to our customer.

And yes, it can be done, but the way it's being done leaves a … bad taste in my mouth. Basically, traffic for the customer's network arrives in our network, hits our router, the router will then encapsulate that network traffic within another IP connection back out out network and through the customer's existing provider to a router (which we set up) at the customer's office.

Yes, we're tunnelling IP packets within IP packets.

There's nothing inherently wrong with that, but it does have all sorts of fun implications for the customer's network traffic (for instance, they also have a wireless router at their office. They could only bring up Google. Any other site would fail to load, or only load a partial page. Turns out that since we're tunnelling IP packets within IP packets, the maximum packet size we can transmit is a bit smaller than normal, only the wireless router couldn't figure this out—I had to manually set the packet size to be about 3% smaller than normal on their wireless router. Why could they get to Google? Probably because Google configured their routers to send small packets).

(Oh, another for instance: it's almost impossible to traceroute to the customer's network, so now it's harder to trackdown networking problems. Oh, and another one I just remember: it adds an additional 11 hops that packets have to travel to reach their final destination.)

Sunday, January 29, 2006

Might as well get free phone service if you're being tapped …

“Here's one more tidbit on wire-taps: They get you free phone service! The feds tapped the phone of the Sisters of Mercy in Washington D.C. because of some anti-war stance or something they took in the 1980s. The good sisters noticed some kind of clicking on the phone at times, and finally decided that someone must have tapped into their phone. Their solution: Don't pay the bill so the phone company will have to shut off the phone. The phone never went dead, and they quit sending them bills! The Feds wouldn't let Ma Bell shut them down, and probably began paying the bills. The sisters talked long and free with their friends across the country!”

I, Cringley—The Falafel Connection

Well … that's one way of getting free phone service.

The article does present a different view of the recent NSA wire taps—that they're less listening in on all the calls and more a “connect-the-dots” by looking at who called who. Defintely an interesting read.


I must apologize to all that read my site via RSS, especially to those that read the LiveJournal feed (for essentially spamming your frends list) last night.

I upgraded mod_blog last night—a huge update. Months ago, I did a near complete rewrite, but was relunctant to put it into production despite testing and using the software on a few other blogs (that aren't mine—they're someone elses). But a few days ago I got an idea for a new feature for my blog, and of course I'm going to want to work with the latest codebase of mod_blog, so …

Only … well … I forgot I made some changes to the RSS templates processing in the program, but forgot to upgrade the actual RSS templates used by my blog. There was also a bug in generating the permanent links (I got a bit overzealous with the formatting), and those two issues caused LiveJournal to have a fit with my feed for a few hours.

It's been ironed out (well, there is still an issue where I have to manually nudge the program whenever I make a new entry) and the RSS feed is working properly now.

Again, I'm sorry.

Monday, January 30, 2006

Calling out the the Lazy Web

Since jwz (and his site is definitely not safe for work often asks the LazyWeb for advice and while I don't think any of my readers may be able to answer this, I might as well give it a shot (before I end up calling tech support and being put on hold indefintely).

I've got a Cisco 2611 with the latest software. It needs to be connected to our office via DSL (on the interface ATM0/0) to our office (which is already set up for DSL, but to require the other side to authenticate). What magical Cisco incantations do I need to get this to work? (and if know how to integrate it with tunnelling, that would be great)


Tuesday, January 31, 2006

I wonder if Bush will wear a gorilla suit for the State of the Union Address …

[Al Qeada is trying to take away your freedom to gorilla-suit! SAVE NATIONAL GORILLA SUIT DAY!]

MacGyver, with the obligatory paper clip

I fixed a communication problem with some scissors, a bit of wire and a paper clip.

No, really.

I got two pieces of equipment communicating by using stuff I was able to find around the office (because that's all I had to work with).

Yesterday, Smirk drops a BayTech DS9 with one DS72 network interface and eight DS30 modules (which control 12 electrical outlets). I was told to test it and configure the device.

So I plug the device into the office network, started up the DHCP server, and powered the DS9. Sure enough, I see the DHCP request come through, it gets an IP address and I try connecting to it.


I scan the device with nmap and yes, it is studiously ignoring all attempts to connect to port TCP port 23 (standard telnet) and UDP port 161 (SNMP). Since it's studiously ignoring the network, my only recourse is to get at it through the serial port.

It's an RJ-45 connector on the DS9, and I have a DE-9 (commonly referred to as DB-9 but DE-9 is the proper designation for the connector) on my workstation. I just so happen to have an RJ-45 to DE-9 adaptor (for the Cisco routers). I may or may not need the NULL modem adaptor, but that's easy enough to figure out.

Only it doesn't work. Straight through, or with the NULL modem adaptor.

I have to reach for the heavy guns. I hunker down and read the manual. It's a good manual, and it gives the exact layout for the cabling between the RJ-45 and the DE-9 that I need to communicate with the DS9. So I go hunting for a continuity tester to see if the Cisco-specific adaptor I have is wired such that I can use it.

Only we don't have a continuity tester. A $5,000 network cable analyzer sure. But something to test the continuity of a single wire? Nada. Not even Dan the Network Engineer has one.

So I carefully break open the Cisco-specific RJ-45 to DE-9 adaptor and lo—it's not wired according to the DS72 manual, so it's useless. So are the other RJ-45 to DE-9 adaptors we have lying around.

So I have to make one.

Only we don't have any spare DE-9 connectors (male or female) lying around, but plenty of RJ-45 connectors. Hmmm …

I grab about a foot length of network cable, crimp an RJ-45 connector on one end, and (carefully) use scisors (since we lacked actual wire strippers) to strip the insulation off the wires at the other end. I then carefully wrapped the exposed wire around a paper clip to make a small coil, and then slipped that coil around the pins of a male-to-male DE-9 adaptor, and plugged that into the serial cable at the back of my workstation. The RJ-45 end of the jury-rigged cable went into the DS9.

The fragile kludge made from scisors, wires and a paper clip (oh, and a crimping tool) worked beautifully and I was able to configure the DS9 (and I kept cursing myself for not having my camera around to document such an engineering feat).

I did find out one thing about the device, and I don't think it's mentioned in the manual (although I was more concerned about the serial connection than anything else) but the device simply drops packets unless they come from a configured SNMP trap host. Good thing to know.

Oops, Part II.

Again, I must apologize for spamming the LiveJournal feed of this blog. I swear, I have the bugs under control this time! I even found why I had to manually nudge the program whenever I made a new entry (because it thought said entry was still in the future, albeit just a second or two in the future, and the software doesn't display entries before its time).

Again, I'm sorry.

Obligatory Picture

[Don't hate me for my sock monkey headphones.]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2017 by Sean Conner. All Rights Reserved.