The Boston Diaries

Sunday, February 07, 2010

More than you care to know about syslog

So I've been learning more than I ever wanted to about the syslog protocol. There's the non-spec that is RFC-3164 that is optimistic in terms of the protocol. Then there's the cleaned-up spec that no one is using that is RFC-5424 (which is quite nice, if a bit over-engineered).

RFC-3164 documents the use of UDP as the transport protocol for the syslog protocol, reading that RFC one gets the impression that one should never actually useUDP as the transport mechanism, least some cracker intercept or change the messages, or worse yet—you lose some packets and get nailed in an Sarbanes—Oxley audit (or even worse still—an ISO-9000 audit—the horror! The horror!).

Well, you could try running the syslog protocol over TCP, but even that isn't good enough for some people, claiming that you can still lose logging information under certain circumstances. No, for reliability you need to add a layer of framing over TCP and wrap the syslog protocol in XML and call it a day.

So far, the only syslog program I've found that even pays RFC-3195 lip service is rsyslogd, and even then, it's receive only and uses its own framing layer over TCP for sending.

I personally haven't seen an issue with using UDP for the syslog protocol. Not only do I relay syslog messages to a centralized server (my desktop box at Chez Boca, so I can watch the stuff in real time) but copies are kept locally (just in case). Also, there have been times when a TCP version (yes, even if I was using RFC 3195 or the lighter RELP) would have failed (at one point, our upstream provider upgraded a firewall that filtered out TCP traffic routed asymetrically and guess what? Our traffic was routed asymetrically; UDP traffic was unaffected and thus in that case, we were able to isolate the issue faster). Even the design of SNMP centered around UDP simply because it was “fire and forget” and thus on a congested network, there was a greater chance of UDP traffic of making it out and accepted than TCP traffic (which requires an acknowledgment that might never make it back).

But in looking over these, I'm struck that a reliable syslog protocol doesn't use SCTP, which has the reliability, ordering and (most importantly, congestion control) of TCP with the message-based semantics of UDP. Heck, for “reliability” SCTP has one feature that neither TCP nor UDP have: either peer can change the IP address used for the session.

For now, I'll just stick with UDP.

Insanity

From
Mark Grosberg <XXXXXXXXXXXXXXXXX>
To
Sean Conner <sean@conman.org>
Subject
Re: Password updated
Date
Tue, 5 Jan 2010 11:24:10 -0500 (EST)
On Tue, 5 Jan 2010, Sean Conner wrote:
What's the cookie2 header for?
I'm so glad you asked. This is almost so good it may cause you to blog about it (actually I figured by the time we were done discussing the insanity of cookies you may have had an insightful blog post anyhow).
I guess after the third cookie spec they figured they kinda sucked at this so they built in an escape. So after much re-re-re-reading of the RFC I think what happens is if you have received a cookie with a $Version that you don't understand you are supposed to just send back a Set-Cookie2: header with $Version="version_this_thing_understands"
It's for future expandability so when we have 10 cookies specs clients and servers will “just work” (at this point I think we both know that statement is about as truthful as “the check is in the mail.”).

Well Mark (and yes, I know, it's been a month), the cookie specs are a paragon of clarity compared to the laughable mess that is syslog protocol specification. Had I been aware of the “informational nature” of RFC-3164 I might not have even started my own homebrew syslogd replacement (network stuff in C, high level logic in Lua).

How loose is the spec?

A program that wishes to use syslog() may select a “facility” the message will be logged under—think of “facility” as a subsystem, like “mail” or “cron” (under Unix, cron runs scheduled tasks on a periodic nature) or “auth” (authorization, or login credentials). Also, each message has a priority (kind of), one of “debug”, “info”, “notice”, “warn”, “err”, “crit” (for critical errors), “alert” (even more critical errors) and “emerg” (basically, the machine is on fire, abandon all hope, etc.). The program using syslog() can also tag each message, usually with its name, and the message itself has no real structure, originally being meant for human consumption.

Now, the syslog protocol, which is used to send the messages to a program that handles these messages, usually named syslogd under Unix, is a text based protocol, and a full RFC-3164 message would look something like:

<87>Feb 07 04:30:00 brevard crond: (root) CMD (run-parts /etc/cron.hourly)

You have the facility and priority (as a single number) in angle brackets, immediately followed by the timestamp, a space and then the name of the machine sending the message, a space and the tag (usually the name of the program on the machine sending the message), a colon, then the message.

And technically, every field is optional! Which makes parsing this a technical challenge. Not only that, but since there never really was a spec, it's easy to find ambiguous messages, such as:

<14>Jan 14 05:53:37 gconfd (spc-25469): Received signal 15, shutting down cleanly

which (per the spec) was sent from the program “(spc-25469)” on machine “gconfd”. Funny thing is, I have no machine called “gconfd” but there does exist a program called gconfd that runs on my machine, running as me, with a process ID of 25496 (fancy that).

I don't even want to talk about /Applications/Windows Media Player/Windows Media Player.app/Contents/MacOS/WindowsMediaPlayer.

It gets even worse. RFC-3164 makes a point in saying that the following is a legal syslog message that has to be processed:

Use the BFG!

Just writing the code to parse this mess took the majority of time, as I kept coming across syslog messages that really weren't.

To work my way out of this mess, if I don't find a proper facility/priority field, I log the raw message (using facility “user” and level “notice”, which is what RFC 3164 says to use in the absence of such information). If there's no timestamp, okay, but if there is one but it's malformed, I log the raw message. I then check for an IP or IPv6 address, as I feel that's really the only sane value to use, then everything else up to a ':' is accepted as the tag value (more or less).

Is it perfect?

No.

But so far, it covers everything I've personally encountered. It will misparse (which is a testcase I pulled from rsyslogd), but not crash, on seeing:

<130> [ERROR] host.example.net 2008-09-23 11-40-22 PST iapp_socket_task.c 399: iappSocketTask: iappRecvPkt returned error

Garbage in, garbage out (also, stuff like this can be checked in the Lua code, as the raw message is available in addition to the parsed message).

Cookies? Insane? Not really. Not when compared to the syslog protocol.

Heaven forbid he ever try to climb a tree (not like it's easy to climb palm trees … )

I'm beginning to think child safety is getting way out of hand. I was driving home from resolving a customer issue (nothing major—just needed to reset a port on a switch—thank you ever so much, Monopolistic Phone Company), driving down the street towards Chez Boca when I saw a kid, maybe four or five years old, wandering about, on foot, close to home (easily within 50′ of the front door), wearing a bicycle helmet! There was no sign of any type of pedal-powered vehicle in the vicinity, although his mother was nearby, sitting on the front lawn watching out for the little kid.

He was wearing a bicycle helmet while walking!

Around his age, I was flying headfirst into ditches on my bicycle sans helmet and the only lasting affect is an inability to spell sertain words correctly. Well, that, and a tendency to say “that” instead of “who.”

He was wearing a bicycle helmet while walking, people!

Then again, I should be amazed he was let outside at all …

Saturday, February 06, 2010

Thinking outside the dusty box

My computer has a rather odd cutout in the front that tends to accumulate dust:

[No, it doesn't glow; I'm just lighting it from behind.]

I've noticed recently that it's accumulated a metric buttload of dust (the shot above is after I've removed quite a bit of it, as shown below). Even Mark remarked how dusty it was in there.

So, the problem: how to go about dusting the thing. It was a thick blanket of dust, and I headed into the garage of Chez Boca thinking I might find a vacuum with a nozzle attachment that would fit in there when I came across the perfect solution:

The Lint Roller!

It's really just a very wide roll of masking tape with the sticky side out. The method is easy: just remove the outer sheet (it's about 6″×4″) and apply it to the dust.

A few lint sheets later and the computer is now dust free.

Wednesday, February 03, 2010

I can't remember the last time I used a printer …

I hate printers.

All I wanted to do was print out a 13 page file using both sides of the paper. It was easy enough to select the odd pages, then the even pages in the printer software. The hard part? Knowing which way to orientate the papers for printing on the second side.

It took me four attempts to get it correct.

Sigh.

So much for saving paper …

Lua just in time

Playing around with Lua is fun, but I've been hearing some good things about LuaJIT, a “just in time” compiler for Lua for the x86 platform (written by a single guy, no less!). Even more amazing, it's literally a drop in replacement for Lua (both the command line interpreter and library).

Okay, I'm willing to give this a try. I download, compile and install it. I then decide to test it using jumble program I wrote in Lua. All I need to do is change one line:

#!/usr/local/bin/lua

to read:

#!/usr/local/bin/luajit

and rerun the program.

Lua version of Jumble program
version	time in seconds
pure Lua	7.74
pure LuaJIT	3.57
Lua + C	2.06
LuaJIT + C	1.70

LuaJIT easily trounces the Lua interpreter without any code changes (other than specifying a different “interpreter”). The versions with C use a C function to sort the letters in the word and while LuaJIT was faster than the Lua + C version, the very fact that I didn't have to modify any code is fantastic! LuaJIT used the very same C code as the Lua version—no changes or recompilations required!

Very neat!

I just relinked my Lua daemon against LuaJIT, just to test it out, and yes, it worked without any changes. I could even reload the scripts on the fly. And incredibly, it's only about 50% bigger than Lua itself.

LuaJIT is one sweet piece of technology.

Perhaps an 80M script isn't that excessive …

Around three months ago, I found a bug in Lua (and yes, it's silly to run an 80M script, but then, I tend to do silly things with programs). I reported the error to the Lua mailing list, and a few days later it was posted as a known bug with a one line patch to fix it.

And yes, I just got around to retesting Lua (with a version that has every patch applied, including the patch for the bug I found) with my 80M script:

[spc]lucy:/tmp/lua>time lua -i show.lua
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> dofile("default.lua")
> os.exit()

real    0m10.964s
user    0m5.880s
sys     0m0.376s
[spc]lucy:/tmp/lua>

Much better.

Monday, February 01, 2010

Yet another Bill Watterson interview found!

Ah, the life of a newspaper cartoonist—how I miss the groupies, drugs and trashed hotel rooms!
But since my “rock star” days, the public attention has faded a lot. In Pop Culture Time, the 1990s were eons ago. There are occasional flare-ups of weirdness, but mostly I just go about my quiet life and do my best to ignore the rest. I'm proud of the strip, enormously grateful for its success, and truly flattered that people still read it, but I wrote “Calvin and Hobbes” in my 30s, and I'm many miles from there.
Via Hacker News, Bill Watterson, creator of beloved 'Calvin and Hobbes' comic strip looks back with no regrets | Living - cleveland.com - cleveland.com

It's not the best Bill Watterson interview (a much better interview was done by Andrews McMeel Publishing) but that answer is just great.

The reason for even linking to this is that any interview with Bill Watterson is a very rare event. Heck, even pictures of the artist are very rare and so far, I think this is the only picture I've seen of him:

[Reminds me of Calvin's dad, or maybe his uncle Max]

Quite the recluse, that Mr. Watterson.

He's also famous of not allowing any Calvin and Hobbes merchandising, although it does appear he relunctantly gave his endorsement for a new product coming out in July:

Monday, January 18, 2010

“Help! I'm trapped in a Chinese fortune cookie factory!”

I crack open the fortune cookie, and, as God is my witness, I read:

[Well, why not? Admit it—you're intrigued.]

And admit it—you're intrigued too!

Saturday, January 16, 2010

“Mawage. Mawage is wot bwings us togeder tooday. Mawage, that bwessed awangment, that dweam wifin a dweam … And wuv, tru wuv, will fowow you foweva …”

Ah, mawage, uh, I mean, marriage. Today my friend Kurt said goodbye to bachelorhood and married his true love.

What else can I say? The bride was beautiful. The groom dashing. The venue had a beautiful view of the Miami skyline, and somewhere, David Caruso is wearing his shades.

Congratulations, Kurt and Amanda!

Friday, January 15, 2010

“Life should NOT be a journey to the grave with the intention of arriving safely in an attractive and well preserved body, but rather to skid in sideways—Chardonnay in one hand—Chocolate in the other—body thoroughly used up, totally worn out and screaming, ‘WOO HOO, What a Ride!’”

My friend Kurt is finally tying the knot tomorrow and of course that can't pass without a bachelor party!

His brother Erik planned a full day of activities and those of us that can take off work, have done so. The first stop on today's debauchery was the Bass Pro Shop for a little archery.

The place is huge. The store itself must be several acres in size and it includes both an indoor shooting range and indoor archery range. I was running a bit late and once I arrived in the store, I had to call Kurt for directions to the archery range (on the second floor, no less!). There I met Kurt, the groom-to-be, Erik, Rich, Kurt's brother-in-law-to-be, Keith and Mike.

We had fun. The various animal targets (an alligator, bear, a wild boar, a few bucks) are on hydraulic lifts and can be raised or lowered by an operator. Given that I haven't shot an arrow since 8^th grade, the bows were remarkably easy to use and we were all mostly able to hit (or just graze) the various tagets.

Gregory arrived just as we were leaving the Bass Pro Shop (he could only get a half-day off from work) for lunch. After a brief discussion, we decided to head to Ernie's Bar B Ques and Lounge (formerly known as “Dirty Ernie's”).

[Life should NOT be a journey to the grave with the intention of arriving safely in an attractive and well preserved body, but rather to skid in sideways—Chardonnay in one hand—Chocolate in the other—body thoroughly used up, totally worn out and screaming, “WOO HOO, What a Ride!”]

The story, as told by Kurt, is that Ernie opened up the place and covered the walls with his sayings on how to live life and whatnot (thus the former name—“Dirty Ernie's”). After a few years of running the place, he apparently got bored, up and left for parts unknown, without even selling the place.

Interesting character.

And good food. And drinks.

Mike warned Kurt that if he passed out tonight, Mike would make sure Kurt ended up with a permanent reminder of the night. It was my idea to make said reminder a tramp stamp. Mike wanted to make it a butterfly, I was leaning more towards My Little Pony, but we still had time to decide.

A few hours later, we arrived at Old Heidelberg for dinner. As this was later in the evening, this meant more friends, and we had the entire back room to ourselves, and two waiters (two friendly fellows by the name of Jeff and Leo). Joining us were Kurt's two other brothers, Neal and Kyle, Russ (who spent the day driving from Tampa), Keener (who spend the day driving from Blountstown, Florida, an even longer drive), Jeff and two other friends of Kurt whose names I didn't catch.

[You'll never find a more wretched hive of scum and villainy]

For the most part, the food was excellent and the deserts—oh—to die for (the Black Forrest Cake I had was indescribably good). After a few hours of dinner and conversation the group headed out to downtown Ft. Lauderdale for a round of bar hopping, but on the way, the evening was having its affect on Kurt and we ended up making several stops on the Bathroom Tour of Ft. Lauderdale™.

We did hit a couple of bars before Kurt felt the need to visit a “gentleman's club.” And while Ft. Lauderdale isn't Lost Wages, what happened there shall remain there. I will, however, make two comments about the “gentleman's club” we visitied:

there were TV screens mounted everywhere and occasionally they would flash “Feel Free To Use Your Credit Card” and “We Have An ATM” (and as Dave Barry says, “I am not making this up”);
the music was loud. No, I ba-da boom boom boom boommeanBa-Da Boom Boom Boom BoomreallyBA-DA BOOM BOOM BOOM BOOMloudBA-DA BOOM BOOM BOOM BOOMandBA-DA BOOM BOOM BOOM BOOMob …BA-DA BOOM… nox …BOOM… ious …BOOMI MEAN SO LOUD THERE WAS A STIFF WIND BLOWING THROUGHT THE PLACE!BOOMWHAT?

Fortunately for Kurt, he never did pass out (then again, how could anyone pass out with music that loud?). Unfortunately for us, we couldn't give him his tramp stamp.