The Boston Diaries

Thursday, November 01, 2007

Fifteen hundred words a day apparently is a problem.

Today is November 1^st.

And that can mean only one thing: the start of National Novel Writing Month.

And like many years gone past I will yet again attempt to write a novel.

Last year (as well as years before) I really didn't have a solid idea going into the month on what to write, so I never got very far. Part of that problem is that I have a hard time writing fiction. While I don't lack for ideas, I do however, lack the discipline to see these ideas through.

I said as much to Spring, and she suggested an idea I can get behind: to write a non-fiction book about a fictitious subject.

It's certainly a better idea than writing 50,000 fictitious words.

We'll see how far I get this year.

Friday, November 02, 2007

Notes on releasing an open source project

I spent the day cleaning up the build process for the greylist daemon as a few people have asked for it. I think the worst aspect of this is writing the gobs of documentation for the program. What I have now is enough for someone familiar with installing non-autoconf packages to get it up and running, but there's still plenty more to write.

Sigh.

Saturday, November 03, 2007

Yup, fifteen hundred words a day is about fourteen hundred a day too many

I'm begining to think that writing 50,000 fictitious words is probably the only way I'll ever complete a novel during National Novel Writing Month.

Spring and I went to a local meetup of fellow Nonowrimoers in Palm Beach County, which clarified my thoughts about Nanowrimo—basically, I'm not up to the task. I just don't have the headspace for fictional writing, nor even non-fictional writing about a fictional subject.

Part of that is lack of discipline; on remaining focused on finishing a task when I've become bored of it, or I find it too difficult and wander off on some other project. Another part is the aformentioned headspace. It's not that I lack ideas—I have plenty of ideas. I just lack the ability to write more than a few hundred words about any of them.

And it certainly didn't help matters that the atmosphere of the meetup was not conducive to writing—what with having to block out the loud conversations about Harry Potter and classroom hijinks.

Sunday, November 04, 2007

A 64bit version of the greylist daemon

I met with Mark (and JeffK) for dinner tonight as payment for helping to get the greylist daemon running on his system. I was glad to do it, if only to keep me honest in my code; Mark's system is a 64-bit version of OpenBSD on a non-Intel platform, and it took a few days (since I didn't have direct access to his systems and we were communicating via email) to get it mostly working (there are apparently a few details I still need to iron out).

Monday, November 05, 2007

And I thought I was being extravagant for using 20 megabytes for the greylist daemon

As if the problems I'm having getting the greylist daemon 64-bit clean and less Linux-centric weren't bad enough, today I find that the Sendmail client just stopped working. I checked the logs, and sure enough, the milter library is bitching up a storm:

Oct 31 17:32:08 XXXX smgl: Greylist Daemon: thread_create() failed: 12, try again
Oct 31 17:32:44 XXXX last message repeated 5 times
Oct 31 17:33:13 XXXX last message repeated 10 times
Oct 31 17:33:17 XXXX smgl: Greylist Daemon, mi_rd_cmd: read returned -1: Connection reset by peer
Oct 31 17:33:17 XXXX smgl: Greylist Daemon, mi_rd_cmd: read returned -1: Connection reset by peer
Oct 31 17:33:17 XXXX smgl: Greylist Daemon: thread_create() failed: 12, try again
Oct 31 17:33:50 XXXX last message repeated 13 times
Oct 31 17:34:52 XXXX last message repeated 28 times
Oct 31 17:35:54 XXXX last message repeated 34 times
Oct 31 17:37:02 XXXX last message repeated 21 times
Oct 31 17:38:05 XXXX last message repeated 7 times
Oct 31 17:39:11 XXXX last message repeated 16 times
Oct 31 17:40:20 XXXX last message repeated 7 times
Oct 31 17:41:31 XXXX last message repeated 8 times
Oct 31 17:41:50 XXXX smgl: Greylist Daemon: thread_create() failed: 12, try again
Nov  1 12:22:53 XXXX smgl: Address: 127.0.0.1:0
Nov  2 19:25:21 XXXX smgl: Greylist Daemon: mi_stop=1
Nov  2 19:27:56 XXXX smc: Address: 0.0.0.0:0

Okay. That mi_stop=1 doesn't look good; let me see what I can find out about that:

On Tue, 27 Jul 2004, Matt Selsky wrote:

Jul 27 09:12:20 clover mimedefang[22489]: [ID 649295 mail.info] MIMEDefang-2.44: mi_stop=1

This is normal.

Regards,

David.

[Mimedefang] mi_stop=1 message in syslog

Lovely! Not only is it normal, but the only reference I found was dated 2004!

There was nothing at all about mi_rd_cmd returning -1, but I did find the following about the thread_create() failed: 12:

On Thu, 28 Oct 2004, Stephane Lentz wrote:

PS: a folk running Linux reported a similar problem but he's not runing RH. On Mandrake/SuSE I've never seen it. How much traffic do you process ? Which hardware ? Try to get some recommendations on system tuning from RH since you're paying $$$$ .

Here's some free advice: On RHEL3, type “ulimit -s”:
$ ulimit -s
10240
So each thread wants 10MB of stack space. That can chew up your RAM pretty quickly. I recommend editing the MIMEDefang startup script and putting:
ulimit -s 2048
just before mimedefang (not the multiplexor!) is invoked.

Right now, the sample red hat script does it only if you have more than 100 slaves, but it should really do it unconditionally.

Regards,

David.

[Mimedefang] thread_create errors

Sweet Jesus! No wonder the Sendmail client sucks up memory like there's no tomorrow! Sheesh!

This doesn't appear to be a reprint of an Onion article—perhaps that's why I find this so frightening

The 23-year-old, who said she had left school without a maths GCSE, said: “On one of my cards it said I had to find temperatures lower than -8. The numbers I uncovered were -6 and -7 so I thought I had won, and so did the woman in the shop. But when she scanned the card the machine said I hadn't.

“I phoned Camelot and they fobbed me off with some story that -6 is higher—not lower—than -8 but I'm not having it.”

Via theferrett, ‘Cool Cash’ card confusion

No wonder the Week ly World News folded—it couldn't compete with the real news.

I wonder what grade this would get at the Harvard Business School?

And speaking of ~~lotteries~~ tax breaks for the smart …

In 1992, a group of Australian investors—led by math whiz Stephan Mandel decided they could literally buy the $27 million jackpot in the Virginia state lottery. Racing against the clock, they bought 5.6 million of a total 7.1 million possible combinations.

The Australians walked away with the only winning ticket—and $27 million.

Big lotto jackpot not what it seems

Actually, I'm surprised this hasn't happened more often (or maybe it does and is underreported). In our own little lovely state of Florida, our little state lottery has a 22,957,480:1 odds of winning, so it would seem that somebody would try it, once the pot has grown to, maybe, three times that amount, which does happen, to make it profitable, if maybe a bit logistically daunting.

I would also expect that the Australians probably did a bit better than the $27 million, because lesser combinations (like 5 out of 6) also pay out.

Tuesday, November 06, 2007

A shot to the arm

[Back many years ago I wrote a humor column for the FAU newspaper (which doesn't exist as I knew it, but that's a story for another time), in which half the time I took a small incident in my life but put a highly fictional spin to it. Perhaps that's what I really need to do—get back to that gonzo mindset and relive my early ~~childhood~~ 20s as a semi-fictional writer. Or something like that.]

[Oh, and I forgot—you have been warned.]

Concerned for my health, Bunny thought it prudent that I get a flu shot, and she knows me well enough to know that I wouldn't willingly go get one on my own. I hate shots. I hate needles. It's probably the only thing that kept me from becoming a heroine junkie.

Well, that, and the relative lack of non-sequential US $100 bills.

But mostly it was the needles.

I was touched by her concern, but felt that dragging me kicking and screaming into the clinic by my ears was uncalled for; her .357 would have certainly made the point clear and been less painful [But as she's quick to point out to the writer, she doesn't have a concealed weapons permit. Yet. —Editor]. I will say that to the nurses' credit, they didn't bat an eye as we came in screaming; they just shoved forms our way and turned to the only other customer there, an older, chain smoking gentleman complaining about an upper respiratory problem, and chided him to put out those cigarettes.

I refused to fill out the forms.

Bunny refused to let go my ear, a bit harder this time.

After a few minutes of this painful stalemate, I compromised. I filled out the form, but left the Social Security field blank. I'm such the rebel.

I don't remember much past that though. I think I tossed the forms back at the nurses, then made a dash towards the door. I either ran into the older chain-smoking gentleman with an upper respiratory problem who was giving one of the nurses a piece of his mind with the most graphic of language, or Bunny body tackled me. In any case, the world quickly turned dark as I experienced sudden deceleration trauma.

I awoke to a piercing pain in my right arm. The nurse was grinning as she shoved a bit harder. I countered with a piercing shriek. She countered with shoving the syringe in hard. I countered by blacking out.

“There,” said Bunny when I finally awoke. I found myself lying on the floor, looking up. “That wasn't so bad, was it?”

Wednesday, November 07, 2007

Named

Since Smirk helped to fund the development of the greylist daemon he had a hand in naming the actual product, and we decided upon X-Grey. The site will be up in a few days, and then it will be officially released to the public.

Woot.

Thursday, November 08, 2007

Notes on an Emergency Room Conversation

“Clear the way!” I said, rushing into the Emergency room with The Younger slung over my shoulder. “Medic! Cortisone stat!” I plopped The Younger down on the counter in front of the on-duty nurse.

“Who are you, and what's wrong with him?”

“You!” I pointed at some random medical personnel who happened to be walking by, “500ml of saline solution, stat! This boy is in real need of medical attention. Take a look,” I said, pointing to The Younger's hand. The nurse at the front desk picked up The Younger's hand and looked at it.

“I don't see anything wrong.”

“What are you, blind? The other hand! Where's that saline solution? Where's the cortisone?”

The nurse picked up the other hand and peer closely. “I still don't see anything.” I pointed to a spot on The Younger's hand. “What is that? A mosquito bite?”

“That,” I said, “is a puncture wound from a staple.”

The nurse blinked at me. Loudly. The whole room was filled with the sound of the nurse's blinking. “A staple wound.”

“My God nurse! How can you stand there like that? This boy needs medical attention! Stat! Where is that cortisone?” I started looking around the room for anyone to order around.

“How old is he?”

“Eleven,” I said.

“Twelve,” said The Younger, with an exasperated tone to his voice as if he's been constantly correcting adults in his age for two months.

“Twelve,” I said. “You!” I pointed to another random medic. “Warm up the CAT scanner!”

“He attends middle school, right?”

“Have you yet to take your Hippocratic Oath? This boy needs immediate medical attention.”

“Yes,” said The Younger.

“The only thing he needs is maybe a tetanus shot—”

“You! Tetanus shot! Stat!” The random medic I ordered just looked at me rather funny and continued on her way.

“—and if he's in middle school, then he's already had his tetanus shot. It's part of his DTP shot he needed to get into middle school.” The nurse looked rather pissed.

“…”

“Anything else I can do for you or the kid?”

“…”

“Thank you. Have a nice day.”

Friday, November 09, 2007

I hate it when real work intrudes at work

I had planned for a nice quiet day at The Office, one where I could work on getting the website for X-Gray but nooooooooooooooooo! Smirk had other ideas for me, like actual work work, which involved moving several websites from a shared environment (running under Insipid) to individual virtual servers (each running Badminton).

Nothing like having to reverse engineer the setup from one <shudder> control panel to another <shudder> control panel.

Sigh.

Monday, November 12, 2007

This metablogging entry brought to you by the letter B. And by the number 13.

It was a quiet weekend (more like comatoast since I didn't bother to post anything) and even today was rather quiet (well, other than moving websites). I started to write something about string concatenation in Ruby and that if one method is frowned upon, why does it even exist, but I just wasn't in the mood to even bother.

Heck, I havn't been in the mood to post in the past few days.

In other news, I got another testimonial about X-Grey:

From

kelly@XXXXXXX

To

Sean Conner <sean@conman.org>

Subject

Re: Greylist daemon …

Date

Mon, 12 Nov 2007 14:06:12 -0500 (EST)

So far, NOTHING (spamwise). I don't know what I'm going to do with all this free time not deleting spam! (pause 1.5 hours for work interruption)

heh … Thanks …

plus some other feedback (some small bugs, omissions from the installation documentation). So far, everybody who's used X-Grey has been happy with it.

What next? That the Mona Lisa is really a self portrait?

ROME, Italy (AP)—It's a new Da Vinci code, but this time it could be for real.

An Italian musician and computer technician claims to have uncovered musical notes encoded in Leonardo Da Vinci's “Last Supper,” raising the possibility that the Renaissance genius might have left behind a somber composition to accompany the scene depicted in the 15th-century wall painting.

Via Instapundit, Italian musician uncovers hidden music in Da Vinci's ‘Last Supper’

For Bunny, who likes both music and The Da Vinci Code.

Tuesday, November 13, 2007

X-Grey

I'll be the first to admit it's not pretty, and it's rather concise, but in order to get this thing out of the door, X-Grey is officially released.

Wednesday, November 14, 2007

Notes on a possible way to store tuples in a greylist implementation

I was checking some other greylist implementations when I came across an implementation that only stores a 32-bit hash instead of the full tuple. The idea sounded intriguing, and since I'm already calculating a 32-bit CRC as part of my protocol, I thought it might be interesting to see just how effective it might be.

Out of 499,846 unique tuples checked, only 28 had duplicate CRCs. I think I could live with that; I could then store 32× the number of tuples in the same amount of memory. Of course, I lose the ability to see the actual tuples, but for a huge ISP or webhosting company trying to deal with an insane amount of email, that is certainly a viable option (hint, hint—Rob, you listening?)

Thursday, November 15, 2007

I've blogged for so long, I'm beginning to forget what I've blogged about

Heh. Here I was, getting ready to post about The IDE Divide when, searching for past entries about my dislike of IDEs, I found out I already did (three years ago!).

This time, I definitely identified with being a language maven, but I can certainly understand the lure of being a tool maven. When programming, it's not uncommon for me to have half a dozen (or more) terms open, editing code, viewing a few header files, a few man pages open, and a command line sitting ready for compiling. I'm not much on syntax highlighting, but automatic code completion would definitly be nice, or perhaps a form of running commentary on the code.

Friday, November 16, 2007

It seems that the Event Horizon for Computer History is somewhere around six minutes

So I'm reading Reddit and come across an article about the increasing bloat in Microsoft applications. Nothing terribly new there, but this bit:

The Stone Age

Back in 1999, when I was working as an advisor to Intel's Desktop Architecture Labs (DAL), I remember how thrilled we all were to get our hands of Windows 2000 and Office 2000. Finally, a version of the Windows/Office stack that could leverage all of the desktop horsepower we were building in to the next generation Pentium 4 platform …

First-off, let me characterize the state-of-the-art at the time. The Pentium 4 CPU was about to be unveiled and the standard configuration in our test labs was a single-CPU system with 128MB of RDRAM and an IDE hard disk. While a joke by today's standards, this was considered a true power-user configuration suitable for heavy number- crunching or even lightweight engineering workstation applications.

What Intel Giveth, Microsoft Taketh Away

has me going nuclear.

Stone Age? In 1999?

It's 1988 and I'm given an account on the university VAX, sharing the CPU with 50 other people on a system that might have had 4MB of RAM and a few hundred megabytes of disk space (we were only allowed five minutes of CPU time per day, which was enough for regular usage, although a friend of mine did manage to blow through that limit regularly by playing a version of Space Invaders he wrote for it).

It's 1984, and for my birthday (and Christmas of 1983—given that my birthday is two weeks after) I received a Color Computer 2, running at a heart stopping 889kHz with a whopping 16KB of memory and a highly advanced means of block storage—the cassette recorder (which recorded data at a breathtaking speed of 1500 baud).

Methinks the author of the above article is in desperate need of a clue-by four.

Stone age my XXX.

Saturday, November 17, 2007

Apparently Snopes didn't have anything to say on this topic

I just got the following from Bunny via email, who got it from someone else via email, who got it from someone else via email …

I didn't know this! Did you?

I have been driving for over forty years. One would think I would have noticed the little secret on my dashboard that was staring me right in the face the whole time. I didn't and I bet you probably haven't either.

Quick question, which side of your car is your gas tank on? If you are anything like me, you probably can't remember right away. My solution is to uncomfortably stick my head out the window, strain my neck and look. If you don't do this in your own car you definitely have done it in a borrowed or rental car.

Well, ladies and gentlemen, I'm going to share with you my little secret so you will no longer look like Ace Ventura on your way to the gas station or put your neck at risk of discomfort or injury.

If you look at your gas guage, you will see a small icon of a gas pump. The handle of the gas pump will extend out on either the left or right side of the pump. If your tank is on the left, the handle will be on the left. If your tank is on the right, the handle will be on the right (see photo above). It is that simple!

I don't know how you feel right now but when I found out this morning I felt cheated!

Why don't the dealers share such important information with car buyers? I don't understand why this isn't in the driver's manual? I don't get why any mechanic I have ever been too or know has even thought of mentioning this to me? The only possible explanation can be that all these people probably don't even know!

Go out and share the world's best kept auto secret with your friends as this information is way too important to be kept secret.

So I went out to Lake Lumina (as my car is known—and it's a long story why it's known as Lake Lumina, which is for another time) and checked. Lo' and behold—it was false.

But! It could still work. Even though the gas tank on Lake Lumina is on the left side, and the gas pump printed on my gas gauge with the handle on the right side, the gas pump is oriented the correct way were I to drive up to one.

So it could be that the gas pump is drawn to show the orientation of the pump as you drive up to it. I'll have to check more dash boards to be sure though.

Sunday, November 18, 2007

Nine Inch Noëls

The thought of singing Head Like A Hole in the style of the Chairman of the Board has always been amusing to me, but now I've found something that's just as amusing, yet actually exists—Nine Inch Noëls, a medly of classic Christmas songs sung with Nine Inch Nails lyrics (link via Joey deVilla).

And yes, it is just as funny as it is wrong.

Monday, November 19, 2007

Burning down the house

I'm feeling much better today.

Back on Friday I first felt the inklings of a cold inching its way into my system, and I began a schedule of heavy vitamin-C and a sea of fluids, followed by heavy sleep, and that seems to have done the trick.

But it wasn't all chicken soup and tissues. On Sunday, groggy from having just gotten up (around 7:00 pm), I went to the kitchen to make some tea and soup. I turned on the wrong burner (easy enough to do) and almost burned down Casa New Jersey when a stove cover (these thin metal lids that cover stove burners when not in use) caught fire. Fortunately, Wlofie caught it in time.

Notes on the Great Stove Top Fire Incident of 2007

On second thought, maybe I'm not 100% yet. In rereading the previous entry, I realized I should have put a gonzonian fictional spin on the Great Stove Top Fire Incident of 2007.

Ah well … I'm ~~feeling too lazy~~ still recovering from my near ~~death~~ cold experience.

Tuesday, November 20, 2007

Little green bunnies

“Alba”, the green fluorescent bunny, is an albino rabbit. This means that, since she has no skin pigment, under ordinary environmental conditions she is completely white with pink eyes. Alba is not green all the time. She only glows when illuminated with the correct light. When (and only when) illuminated with blue light (maximum excitation at 488 nm), she glows with a bright green light (maximum emission at 509 nm). She was created with EGFP, an enhanced version (i.e., a synthetic mutation) of the original wild-type green fluorescent gene found in the jellyfish Aequorea Victoria. EGFP gives about two orders of magnitude greater fluorescence in mammalian cells (including human cells) than the original jellyfish gene.

Alba, the fluorescent bunny

Ah, leave it to the French to concoct a gazillion-word manifesto on the artistic and social æsthetics of mutant green bunnies in French culture (and I don't mean a yogurt based sauce either).

And oh, by the way, the mutant green bunny actually exists.

Testing a reverse captcha

Back when you could sign up to get email notifications of updates around here, I noticed some spammers were attempting to abuse the script, and after playing around with it, I decided to scrap that feature, but later mused on using a reverse captcha to keep spammers at bay.

But while I dont' have comments, I do invite comments on another website I control. And that form is now getting spammed (and like my former Obligatory Email Notification, the only one who sees anything is me).

Heavily.

And when they're not hawking 3″ mortgage extensions, I get incomprehensible stuff like:

Name

embohette

Email

gorgiovgit@list.ru

Comment

<a href="http://salihome.info/show/index.html">Three hip-hop artists have key parts in American Gangster.</a><br> <a href="http://oscines.cn/myspace-layout/index.html">Andre 3000 takes a turn with Charlize Theron in Battle in Seattle.</a><br> <a href="http://myspace-layout.tripod.com/">Robert De Niro and Al Pacino reunite next year in the crime drama…</a><br> <a href="http://rigolu.cn/sex-portal/main.html">MORE big screen… </a>

Submit

Panic

So I decided to put my “reverse captcha” theory to the test. I added a non-displaying <TEXTAREA> to the form, and if it's changed in any way, I ignore the submission. I'm curious to see how long it takes the spammers to adapt, if any bother.

Monday, November 26, 2007

A little lapse in posting

Wow, a week has gone by without a post.

What happened?

Um … a bloggers strike … yea … that's it. The local Blogger Chapter #34 went on strike. You know how it is …

The Secret Temple (that's not so secret anymore)

Here, 100ft down and hidden from public view, lies an astonishing secret one that has drawn comparisons with the fabled city of Atlantis and has been dubbed “the Eighth Wonder of the World” by the Italian government.

For weaving their way underneath the hillside are nine ornate temples, on five levels, whose scale and opulence take the breath away.

Constructed like a three-dimensional book, narrating the history of humanity, they are linked by hundreds of metres of richly decorated tunnels and occupy almost 300,000 cubic feet—Big Ben is 15,000 cubic feet.

…

Indeed, the Italian government was not even aware of their existence until a few years ago.

But the “Temples of Damanhur” are not the great legacy of some long-lost civilisation, they are the work of a 57-year-old former insurance broker from northern Italy who, inspired by a childhood vision, began digging into the rock.

Via Instapundit, Eighth wonder of the world? The stunning temples secretly carved out below ground by “paranormal” eccentric

It's amazing what Oberto Airaudi has been able to accomplish, unseen, in thirty years of work.

Musings on high volume email servers and X-Grey, the greylist daemon

On Saturday, I bumped into Rob at a “After Thanksgiving Party” and we discussed the use of X-Grey at Negiyo, at least, those parts of Negiyo email that Rob helps to manage.

The code, as is, won't work with their setup. First problem, the sheer volume of email—something like 100,000 connections per second. These are fed through two load balancers and farmed out to about 100 servers, so each server is responsible for 10,000 connections per second. While I suspect X-Grey can handle 10,000 connections per second, the major problem are the load balancers—there's just no guarantee that the load balancers will be consistent on which machine they send the connection to.

For instance, we have some machine, on IP address 10.20.30.40 sending an email from alice@example.net to bob@negiyo.com. The load balancer will send that to server A, which doesn't find the tuple [10.20.30.40 , alice@example.net , bob@negiyo.com], stores it for later reference, and sends back “try again later.” Later, the machine at 10.20.30.40 tries sending the email again, only this time, the load balancer sends the connection to server B, which doesn't find the tuple, stores it, and sends back “try again later.” Lather, rinse, repeat until the sender gives up, or the load balancer manages to send the traffic to a machine that actually has the tuple stored.

There's just no way of knowing which server the load balancer will send the traffic to. So, we point all the servers to a single greylist server, which now has to handle 100,000 requests per second. Okay, so assuming X-Grey can handle that load (it's a real beefy box on a fat pipe), and given that we store greylisted tuples for six hours … carry the one … 2,160,000,000 tuples.

Blink.

Okay, now that I'm actually doing the math instead of sitting around in a comfortable chair listening to Rob while chowing down on turkey and stuffing, I find it rather difficult to believe that Negiyo is getting around 8½ billion emails per day—even a billion per day is stretching my credibility. The worst we get at The Company is 8 per second, with an average hovering around 1.4 (or 122,540 per day, which I calculated twice, using two different statistics that are recorded). More believable is 100,000 per hour (or even up to 1,000,000 per hour, which is 11 emails per second).

I'll have to get back with Rob on this …

Guerrillas clandestinely time their movement in Paris clock

Four members of an underground “cultural guerrilla” movement known as the Untergunther, whose purpose is to restore France's cultural heritage, were cleared on Friday of breaking into the 18th-century monument in a plot worthy of Dan Brown or Umberto Eco.

For a year from September 2005, under the nose of the Panthéon's unsuspecting security officials, a group of intrepid "illegal restorers" set up a secret workshop and lounge in a cavity under the building's famous dome. Under the supervision of group member Jean-Baptiste Viot, a professional clockmaker, they pieced apart and repaired the antique clock that had been left to rust in the building since the 1960s. Only when their clandestine revamp of the elaborate timepiece had been completed did they reveal themselves.

Via tryss, Undercover restorers fix Paris landmark's clock

And speaking of clandestine construction, what's up with the Europeans?

Tuesday, November 27, 2007

Cognitive dissonance

Intellectually, I know that The Holidays are not over yet, and that if anything, they're just now picking up steam, but to me, it feels like it's already over. Normally, this would be a Good Thing™ but the cognitive dissonance I'm experiencing is annoying.

Here it is, just shy of December, and it's still hot. Okay, it's not August “so hot the asphalt is boiling” hot, but it's still warm enough for natives to still be wearing tee-shirts and shorts. There are precious little decorations littering the landscape, and I've yet to see any palm trees smothered in lights.

It just doesn't feel like The Holidays (granted, when I first moved down here it didn't feel like The Holidays then either, what with trees still having leaves—green leaves at that, and no snow, and it wasn't freezing during the day). And I have to wonder why I feel so cheated.

Perhaps because it's still hot.

I think we need to get Al Gore down here—oh wait, gotta go … Bunny wants to shop for Christmas trees …

Mind the announcer

It's too bad that Emma Clark is no longer the voice of the London Underground (link via Flutterby). Some of her announcements are rather amusing.

Wednesday, November 28, 2007

Stupid multithreaded benchmarks

Dan Lyke ran a stupid benchmark—just incrementing a variable to a billion, one just flat out, and one between some pthreads locking primitives.

I wanted to see what stupid results I would get if I used spin locks. First, the relevant bit of code (used nasm to compile it under a 2.6GHz dual-core Pentium running Linux 2.6):

		bits	32
		global	t1
		global	t2

		section	.data
gv		dd	0
glock		dd	0

		section	.text

		align	16
t1:		mov	eax,[gv]
		inc	eax
		mov	[gv],eax
		cmp	eax,1000000000
		jl	t1

		; ret

	; the following implements _exit()
	; for the multithreaded version

		xor	ebx,ebx
		mov	eax,252
		int	$80
		mov	eax,1
		int	$80
		hlt

		align	16
t2:		mov	al,1
t2.wait:	xchg	al,[glock]
		or	al,al
		jne	t2.wait
t2.here:	mov	eax,[gv]
		inc	eax
		mov	[gv],eax
		mov	byte [glock],0
		cmp	eax,1000000000
		jl	t2
	
		;ret
	
		xor	ebx,ebx
		mov	eax,252
		int	$80
		mov	eax,1
		int	$80
		hlt

Straightforward implementations here. t1() is the straight through counting routine, while t2() is the one with the spin lock. Running single threaded yielded these results:

Counting, single threaded
routine	time to execute
`t1()`	2.454s
`t2()`	39.752s

While I expected the spin lock to be faster than the pthread locking, I wasn't expecting it to be this slow. But maybe, just maybe, I'll get some of that speed back by running dual threads. At the very least, it should be a bit faster than single core, right?

Right?

Bueller? Bueller?

Counting, dual-threaded
routine	time to execute
`t1()`	0m10.334s
`t2()`	2m31.307s

Um …

Wow.

I didn't expect spinlocks to be so expensive.

Ouch.

Thursday, November 29, 2007

More stupid multithreaded benchmarks

The other interesting thing to think about: at what point do we beat the same algorithm in C, and how hard would it be to parallelise the algorithm in C with pthreads …

Use those extra cores and beat C today! (Parallel Haskell redux)

Since I'm doing stupid benchmarks anyway, why not this? I didn't use pthreads though—I haven't used that particular API in about a decade, and even then, I wasn't thrilled with it.

Nope. Instead I used a Linux specific call—clone(), wrote a small function to wait for all the threads that were created, and basically spent about five minutes making a parallelized C-version of the Fibonacci sequence. Granted, I had to do the parallelization explicitly, but it wasn't that difficult to do.

The hard part was finding a quad-core box to test this on. Fortunately, I know we have one at The Company (shhh—don't tell Smirk) and that's where I spent most of my time on this—locating our single quad-core machine I could test this on.

But find it I did. And yes, the program does run faster the more cores that are thrown at it, but it's not a linear speed up:

A Parallelized Fibonacci Sequence calculation program
# cores	Runtime
1	0m31.468s
2	0m19.521s
4	0m17.234s

It's interesting that it appears to be bottoming out rather quickly (and these are average times over a few runs).

Friday, November 30, 2007

Obsessing over stupid benchmarks (and you thought my obsessing over a greylist daemon was bad … )

I'm still obsessing over stupid benchmarks. This time, I was curious about the overhead of calling functions and system calls, to simulate a typical locking scenario, like:

int gv;

void t(void)
{
  while(gv < 1000000000)
  {
    lock();
    gv++;
    unlock();
  }
}

So, I wrote some test code:

int gv

void null(void)
{
}

void t1(void)
{
  while(gc < 1000000000)
  {
    null();
    gv++;
    null();
  }
}

void t2(void)
{
  pid_t me;

  me = getpid();
  while(gc < 1000000000)
  {
    kill(me,0);
    gv++;
    kill(me,0);
  }
}

The first test is pretty simple—it just tests the function call overhead, and that particular piece of code takes 9 seconds (give or take a few tenths of a second) to run. But let me explain about the second test, and therein lies the interesting bit of this tale.

I wanted to test the system call overhead, and thus, I needed a system call that did very little. I found this bit in the kill() manpage:

If sig is 0, then no signal is sent, but error checking is still performed.

That's about as close to a null system call as I could find. No real bad side effects, and I don't particularly care about the return code (since this is just measuring overhead). So I ran that bit of code. Fifteen minutes later it finished.

Fifteen minutes.

Either system calls are slow, or that particular system call does a vast amount of nothing. Curious, I delved into that part of the kernel source (I'm running all this under Linux, by the way) when I came across this bit of kernel code:

asmlinkage long sys_getpid(void)
{
	return current->tgid;
}

Now, I knew that getpid() was a system call, but I didn't realize it was a system call. What I mean by that is, even though it's documented in section 2 of the manual (usually reserved for system calls), it may not actually call into the kernel. For instance, alarm() is documented in section 2, but it's a library call that eventually calls other system calls. And I thought that getpid() was a library call that just returned a value (the current process ID) stored in userspace (which makes sense to me—it's not like it changes or anything as the process runs). I didn't expect it to be an actual system call.

Well then.

And it's about as low overhead of a system call as you can get.

So I rewrote t2():

void t2(void)
{
  while(gc < 1000000000)
  {
    getpid();
    gv++;
    getpid();
  }
}

And the runtime dropped to 17 seconds.

Okay, much better. Not quite twice the function call overhead. But what the heck was kill() doing that warranted fifteen minutes? I checked the Linux source code and it didn't appear to be that much. What the heck was going on?

So what's the actual code to getpid()? Was it perhaps making the system call once and storing the result? Time to check the code to getpid(). Easiest way to do that was to statically compile the test program, and disassemble it (using objdump):

0804e380 <__getpid>:
 804e380:       b8 14 00 00 00          mov    $0x14,%eax
 804e385:       cd 80                   int    $0x80
 804e387:       c3                      ret

Okay. Not much to it. What if I make the system call directly then? And here, I resorted to assembly (since the inline assembly directives for gcc make no sense to me whatsoever):

t41:	mov	eax,20
	int	$80    
	mov	eax,[gv]
	inc	eax
	mov	[gv],eax
	mov	eax,20
	int	$80
	mov	eax,[gv]
	cmp	eax,1000000000
	jl	t41
	ret

(oh, the reason the code looks a bit different? objdump uses AT&T syntax, while nasm uses Intel syntax)

It was a bit surprising to find myself waiting for 11 minutes for this bit of code to run.

Okay, that's more on par with using the system call kill(), but that doesn't explain why my version took 66 times longer. Perhaps it's a CPU pipeline issue? Perhaps it is faster to call a function that invokes a system call than it is to inline the actual system call. So I'll called the function getpid():

	extern	__getpid

t42:	call	__getpid
	inc	dword [gv]
	call	__getpid
	cmp	dword [gv],1000000000
	jl	t42
	ret

Back to 17 seconds. Okay, what if I supply my own wrapper function around the getpid() system call?

t43:            call    getpid
                inc     dword [gv]
                call    getpid
                cmp     dword [gv],1000000000
                jl      t43
                ret

getpid:         mov     eax,20
                int     $80
                ret

And just to make sure, check with objdump:

080484b0 <getpid>:
 80484b0:       b8 14 00 00 00          mov    $0x14,%eax
 80484b5:       cd 80                   int    $0x80
 80484b7:       c3                      ret

 ...

0804e420 <__getpid>:
 804e420:       b8 14 00 00 00          mov    $0x14,%eax
 804e425:       cd 80                   int    $0x80
 804e427:       c3                      ret

So, other than the memory location of the routine (and the name, but the name isn't part of the executable) they're the same. So why does my version of the getpid() function take 11 minutes to run? Something odd is going on somewhere.

I then wrote a version of getpid() that caches the return value from the first call (and this time, I made it callable by C functions):

		global	my_getpid

my_getpid:	mov	eax,[pid]
		or	eax,eax
		jz	.10
		ret
.10		mov	eax,20
		int	$80
		mov	[pid],eax
		ret

Two tests, one calling this routine from C, another test calling this routine from another assembly routine. And both take around 10 seconds to run, which is just a shade slower than calling a null routine.

But calling __getpid() (the system supplied C library routine that calls the system call) is faster than calling getpid() (the function I wrote to call the system call), even though they're identical functions.

Is there something else going on that I'm not aware of?

Weird …

Obligatory Death of Satire Links

The Stone Age

Obligatory Picture

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous