The Boston Diaries

Monday, October 11, 2010

Let me 'splain … no, there is too much. Let me sum up

Back in April M (and until he states otherwise, he shall remain M) brought me on as a consultant at The Corporation (not to be confused with The Company, where I work with Smirk and P) to do help profile “Project Wolowizard” (M's name for the project (which deals with telephony communications protocols) and way better than any name I came up with for use on my blog) which later lead to writing a test harness (in C, C++ and Lua) and in late August lead to full time employment with The Corporation (they made an offer I couldn't refuse).

Which is good, because we (“we” being The Company) were let go from “Project: SocialSpace 2.0” due to an internal disagreement high up on the hiring company over the direction of the website. And shortly after that, Smirk was also hired by The Corporation to help with documenation and server ops. P declined to work with The Corporation; the Ft. Lauderdale Office being too far for him to work, so P still works for The Company (along with Smirk amd me, which the Corporation has no problems with).

So between “Project: SocialSpace 2.0” (until it blew up in late August/early September) and “Project: Wolowizard” (still ongoing for the foreseeable future) I've been too busy (okay, laziness has something to do with it as well) to update the ol' blog here. But now that I've settled into a new work groove, I hope things will improve around here.

Unclear on the concept of public and private community strings

Okay, what prompted me to update the blog was an incident that happened at The Corporation.

Like I mentioned, we're working on “Project: Wolowizard,” which involves dealing with SS7, a telephony-based communications protocol. To handle SS7, The Corporation licensed The Protocol Stack From Hell™. Now, given the amount of money that exchanged hands, one would think we would get decent documentation, source code and even technical support, but no, no source code, documentation that is technically correct but completely useless and what can only be described as “obstinate technical support.”

So for the past two weeks Smirk has been tasked with setting up The Protocol Stack From Hell™ to respond to SNMP queries. The technically correct but completely useless documentation indicated that it was a simple matter to set up SNMP support; just configure the IP address, UDP port and community string (really a password) in a few files and it should “just work.”

I wouldn't be mentioning it here if it “just worked.” Not only did The Protocol Stack From Hell™ fail to reply to SNMP queries, but it wasn't even opening up the UDP port. For two weeks, Smirk, R (the office manager in the Ft. Lauderdale Office of The Corporation) and a few others from the Main Office of The Corporation (which is in Seattle, Washington—nice because they're three hours behind us) were going back and forth with the obstinate technical support. Things were going nowhere slowly.

Then R suggested that Smirk set the SNMP community string to “public”, which is a typical default setting for SNMP. Of course that worked. As an off-handed remark, I suggested that Smirk should try “private”—the other, “more secure” community string that is commonly used.

That worked too.

But not the community string we wanted to use.

Head. Meet desk.

(For the record, The Protocol Stack From Hell was selected way before R, who runs the Ft. Lauderdale Office of The Corporation, even worked for The Corporation)

I mean, yes, there certainly was some braindeath in the “Project: SocialSpace 2.0” but at least it worked even if you could measure its speed in bits per eon. This wonderous feature of The Protocol Stack From Hell™ is almost criminal in its action.

Sheesh.

Wednesday, October 13, 2010

Feeding your inner carnivore

All I can say is … expense accounts rock!

A few members of upper management from The Main Office of The Corporation arrived at The Ft. Lauderdale Office of The Corporation to check things out. And because they're upper managment, they pretty much get to expense anything they want, so they took the Ft. Lauderdale Office (meaning: us) out to dinner to a Brazillian steakhouse called Chima.

There is no menu. Instead, you just flip a small disk to the side indicating you want food; then you are inundated with men carrying large hunks of roast beast offering you slices of various cuts of beef, pork or lamb. It's a never ending river of meat; vegetarians need not apply. To stem the rising tide, just flip the disk over to the other side. You can keep doing this as long as you want. As much as you want.

It's insane.

And insanely good.

It also appears to be one of those “if you have to ask, you can't afford it” type places.

Thursday, October 14, 2010

Beware of feeding your inner carnivore

Uhg.

I ate way too much. I should have skipped lunch yesterday had I known about Chima.

I should have also kept the small disk on the “no” side more often.

The food was great going down. It wasn't so much fun coming back up.

Twice.

I emailed in sick, slept for nearly twelve hours (that is, after … um … cleansing my system) and spent the rest of the day recovering.

Saturday, October 16, 2010

“Red”

Bunny and I went to see “Red” tonight and yes, it was worth it. It's an action comedy film about a retired CIA agent (played by Bruce Willis, who is “Retired but Extremely Dangerous”—thus the name of the film) trying to lead a normal life who suddenly finds himself running for his life while trying to protect an innocent love interest (played by Mary-Louise Parker).

The cast is outstanding; John Malkovich playing a paranoid conspiracy theorist who's fun to watch on the screen, the always good Morgan Freeman, still sexy Helen Mirren and Richard Dreyfuss in a small but crucial role.

The film (thankfully) keeps its humor level throughout the film and it manages to keep the action sequences thrilling to watch. It's not high art, nor does it pretent to be anything other than a comedy action film. And for what it is, it's quite good.

Tuesday, October 19, 2010

We were experiencing technical difficulties …

I finally get back into the blogging mode when the server my site (and email) is on goes offline when its power supply flames out.

Sigh.

It was weird today—I felt lost without access to my server, mainly because of the loss of my personal email (yes, I still run my own email server; yes, I'm masochistic like that) but yes, the website being down was also weighing on me—until yesterday (Monday) I don't think any server my site has been on has been down for more than three or four hours (oh, there have been some times recently when the site has been down for over 24 hours but that's because various processes on the box were inadvertantly killed and I didn't realize it, but that's different than the actual server being offline).

But things seem to be back to normal, and I would like to thank the crew hosting my website for their hard work in getting it back up quicker than I expected.

Your complete boating resource

I received this last week:

From

MXXXXX XXXXX <XXXXXXXXXXXXXXXXXXXXXXX>

To

sean@conman.org

Subject

Website Inquiry

Date

Tue, 12 Oct 2010 15:32:04 +0800

Dear Webmaster,

I recently discovered your "Uhg … sushi for breakfast" page here:

http://boston.conman.org/2002/05/03.4

I wanted to let you know that the “PublixDirect” link on your site points to a website (http://www.publixdirect.com/) that is no longer working. Would you please consider replacing it with a link to my website called Boating.com? It is a resource that provides hundreds of used and new boats for sale, as well as reviews, tips and buying guides for anyone interested in boating.

If you think it would be of use to your visitors, would you please consider adding a link to my website on your page here:

http://boston.conman.org/2002/05/03.4

Or in any of the pages of your site that will be most appropriate.

Here is the HTML link you could add: <a href="http://www.boating.com">Boating.com</a> - the complete boating resource.

Will look forward to hearing from you.

Thanks!

MXXXXX

Boating.com
www.boating.com

My original thought was to post it immediately, but I thought better of it at the time. I mean, it's nice that M pointed out a dead link on my site, but given that that particular entry is eight years old that is to be expected (also, PublixDirect was discontinued in August of 2003).

But I found it really odd to suggest replacing the PublixDirect link with a site about boat sales. It doesn't make sense in the context of the entry and it tells me that M probably has some piece of software that looks for sites with broken links that have a certain page rank and spams any address found on the page asking to replace the dead link.

Nice, except I'm not a link farm. The links in each entry provide (or at least, I hope they provide) a context for what I'm writing about and frankly, having the text “PublixDirect” go to a site about boat sales doesn't serve me, or my readers, any good purpose (hmm … but what to do about links that have since changed since I wrote an entry … but that's another ~~show~~ entry). It's also obvious that M never bothered to read the entry in question or else M wouldn't have bothered to suggest replacing the PublixDirect link with some spam site.

I replied to M (but without quite the snarkiness here) but have yet to receive a reply, so a week later, it's fair game. And heck, if I'm lucky, perhaps this entry will become the complete boating resource; that'll show M.

“NPA” seems to be a popular party these days …

I received the sample ballot in the mail today (okay, technically yesterday but I'm still in “today” mode) and I'm excited to see all the non-Republican/non-Democratic candidates in the eight races I can vote in on November 2^nd. There are eight Republicans, eight Democrats (figures the Republicans would have a .com and the Democrats a .org), one Libertarian (Alex Snitker), one Constitutional Party of Florida (Bernie DeCastro), one for the Tea Party (Ira Chester), two Independance Party candidates (Peter Allen and his running mate John Zanni) and fifteen (count them, fifteen) non-party-affiliated candidates.

I find that wonderful for some reason.

Do I really expect the non-Republicans/non-Democrats to win?

Not really, but a guy can hope, right?

Thursday, October 21, 2010

Maybe this year I'll get a novel out—wouldn't that be novel?

Hey, there! Thanks for your interest in my 8-Week Online Novel-Writing Course. It's designed to coincide with November's National Novel-Writing Month (NaNoWriMo), but includes both pre-drafting and revision instruction as well.

I've been a teacher of creative writing at a major university for more than 8 years, and all through that time I've done independent novel-writing coaching as well as written these dang things myself! When you finish my course, you will have a revised draft of a novel ready to post, sell, share, or revise further. (I offer one final read with comments after your revision!) The only way to be a novelist is to WRITE THAT NOVEL! I will be your cheerleader, drill sergeant, and best friend during this whole process.

TwitLo nger: Hey, there! Thanks for your interest in my 8-Week Online Novel-Writing Course. It's designed to coin

My friend Hoade, college English instructor and zombie expert, is offering this novel-writing course.

And I think I'm going to try it.

The limits of sleeping

Testing “Project: Wolowizard” isn't easy, what with dealing with The Protocol Stack From Hell™ and clarifying which optional fields of certain messages are mandatory, further complicated by the fact that some optional mandatory fields are optional if other optional mandatory fields are included, but there's also a feature that if used, means you may not include an optional mandatory field, but if you do, then you need to set another optional optional field.

It's all very confusing.

But while I'm waiting for the first extended test run to finish (24 hours of continous tests), I decided to investigate a particular phenomenon I've noticed when developing the testing program.

We need a way to test various message rates and one of the easiest is somethine like:

for count = 1 , 1000 do
  endpoint:send(msg)
  sim.sleep(thepausethatrefreshes)
end

If I want to send 10 messages per second, then thepausethatrefreshes will be 0.1 (sleep for a tenth of a second); if I want 1000 messages per second, then thepausethatrefreshes will be 0.001. It's an easy way to dial up or down the number of messages per second sent.

The sim.sleep() function (we're using Lua to script our test scenarios) is really a wrapper around the C function nanosleep(), which, in theory, allows a very fine resolution (to the nanosecond) but in reality is limited to the hardware on the system.

And in my testing, I've found that at best, I can get only about 100 messages per second. I wrote a program to test the limits of nanosleep() and found that on our particular testing platform, the lowest I can go is 0.01 seconds (my Linux system, on the other hand, can go as low as 0.002).

As a second test, I just blasted out messages as fast as possible and was able to get a sustained (that is, actually sent and acknowledged) 1,500 per second (or one every 0.0006 seconds). I have to rethink my approach here. I think what may work is to send groups of messages before pausing a bit.

So, if I wanted to send 1,000 messages per second, I would need to write something like:

function thousand_per_second()
  for pauses = 1,50 do
    for msgs = 1 , 20 do
      endpoint:send(msg)
    end
    sim.sleep(0.02)
  end
end

This sends twenty messages per batch, then pauses 0.02 seconds between batches, to give us approximately 1,000 per second (actually, a bit less due to processing overhead).

Tuesday, October 26, 2010

“Multithreaded programming is hard, let's play Minecraft.”

All I wanted to do was pass some data from one thread to another. I had code to manage a queue (based off the AmigaOS list code), so that wasn't an issue. And while a pthread mutex would ensure exclusive use of the queue, I still had an issue with one thread signaling the other that data was ready.

I took one look at pthread condition variables, didn't understand a word of it, and decided upon a different approach.

The primary issue really wasn't the transferring of data; it was the processing to be done with the data. I'm writing code to drive, let's say, a widget, for “Project: Wolowizard” and part of that test code is simulating, say, a sprocket. When the sprocket gets a request, the original code could return a result or just drop the request. R wanted a way to delay the results, not just drop them. To avoid a redesign of the test program, the easiest solution was to just spawn a separate thread to handle the delayed replies.

So, not only did I have to transfer the request, but signal the other thread that it had a new request to queue up for an X second delay.

My first test approach (a “proof-of-concept”) used a local Unix socket for the transfer. This approach had the benefits of avoiding the use of mutexes and condition variables, and the code was fairly easy to write. Just poll() the local Unix socket for data with a calculated timeout; if you get a timeout, then it's time to send the next delayed result, otherwise it's a new request to queue (ordered by time of delivery) and recaluclate the timeout.

But I found it annoying that the data was copied into and out of kernel space (not that performance is an issue in this particular case; it's more of an annoyance at the extra work involved).

Fortunately, I found sem_timedwait() and sem_post(). I could use sem_timedwait() to wait for requests, much like I used poll(). Some mutexes around the queue (I originally used some other semaphores, but M suggested I stick with mutexes for this) and the code was good to go.

Only it didn't work in the actual program. My “proof-of-concept” worked, but in actual production, the second thread was never notified.

I asked M for help, and through his asking the right questions I suddenly had an idea of what was wrong—one quick change to a test script (the testing program I wrote is driven by Lua) proved my hunch right. Which is when I planted my face in the palm of my hands.

The upshot—my sprocket simulation works in one of two different ways (one method uses The Protocol Stack From Hell™ as a test) and the way I picked (which doesn't use The Protocol Stack From Hell™ and is thus, more “safe,” as both M and I are of the opinion that The Protocol Stack From Hell™ is also subtly buggy) failed to trigger the appropriate semaphore. Worse, had I used the local Unix socket, the whole thing would have worked as intended.

One of these days I'll get this whole multithreaded programming thing down pat.

Hoade won't let me get away with 50,000 fictional words

I'm trying it.

Hoade's online novel writing class (disclaimer: we're “best-friends-forevah”) is really helpful. I think it's the structured nature of the course and the (so far) daily assignments (no writing yet!—that's another week or so away) that are keeping me focused.

And the constant emails yelling for completed assignments aren't hurting either (seekrit message to Hoade: Bunny's birds left extensive comments on my homework and trust me, you don't want to see it).

Maybe this time I'll have more than a pile of incoherent notes.

Saturday, October 30, 2010

There are technical difficulties, and then there are technical difficulties

Nearly two weeks ago my server went down because of a faulty power supply. At the time, I made a joke about the power supply going out in flames.

Last night (technically this morning) the new server went down for a few hours (oh great!) but then I see this in email (forwarded by Smirk):

At approximately 1:49 a.m. EDT, UPS #2 experienced a catastrophic failure resulting in a fire. As a result, a portion of the XXXXXXXXX X datacenter experienced a power interruption that may have impacted some customers.

The Fire department responded to the fire in UPS #2 immediately, arriving on site at 1:57 a.m. EDT. For safety reasons, access to the XXXX XX area of the facility was temporarily restricted.

Oh.

Well then … (memo to self: no more jokes about servers catching fire)