The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Sunday, February 01, 2015

The Jedi sure are manipulative bastards

At the end of Episode III, there are two new babies who are down two viable parents—mom's died of a broken heart (or something) and dad has become a tyrannical right hand to the galaxy's freshly minted dictator. They need homes but, more importantly, they need to be kept safe from interfering influence. Most of the Jedi have been wiped out, and resistance to the Empire is a bad bet when it's on its way in; Palpatine has too many resources, he planned his takeover too carefully.

Yoda and Obi-Wan knew all this coming out of the Clone Wars. They knew that even with the handful of Jedi who survived the Purge at their disposal, they had no chance of taking on the new regime. It was better to let Palpatine get comfortable, to hide away and await a better opening. And considering the power that both Vader and the Emperor could claim mastery over, using Anakin's kids were a good bet; after all, their dad was literally conceived out of raw Force trail mix and some of Shmi Skywalker's genetics. They were bound to have similar power at their disposal, the better to knock off dear old dad with.

Here's the problem with children—they grow up to be fully realized human beings. The Jedi knew what they were doing, taking infants from their parents to indoctrinate them into the old Order. Initiating kids before they could talk, becoming their true family, resulted in “better” Jedi. (Translation: Jedi who do as they're told by the Jedi Council and their mentors.) But Anakin's children were better off being raised out of contact with Jedi. That way, if someone managed to locate Yoda or Obi-Wan, the Only Hopes of the galaxy remained safe and secret.

Via MyFaceGooglePlusSpaceBook, This is Why Obi-Wan Lied to Luke Skywalker About His Father | Tor.com

It's a long article, but it does seem to explain why Obi-Wan outright lied to Luke about his parentage. I never expected the Good Guys to lie, but the Bad Guys? Yeah, I expect them to lie to get ahead, that's what Bad Guys do!

But not the Good Guys! The Good Guys don't manipulate people, don't lie, fight fair, and above all, never act like a vigilante.

Um … are we sure the Empire aren't the good guys? Yeah, they're incompetent, but really? The Bad Guys?

Monday, February 02, 2015

The smallest chess program ever written

Thirty-two years ago, David Horne wrote a chess program for the ZX-81. It didn't play a great game of chess, and you can't castle, capture en passant nor promote a pawn, but it did have one redeeming feature that set it above every other chess program—it took less than 1K of space! The program, in its entirety, is only 672 bytes in size.

But there's a new contender for the smallest chess game (and the same limitations—no castling, no en passant, no promotion) with BootChess, which is an incredible 487 bytes in size!

---

Even a program of a single instruction can have bugs

The previous entry reminded of the following computer joke:

Every program has at least one bug and can be shortened by at least one instruction—from which, by induction, one can deduce that every program can be reduced to one instruction which doesn't work.

Only it's not a joke—it really happened!

Tuesday, February 03, 2015

And I still haven't found what I'm looking for

If I have any text processing to do, I pretty much gravitate towards using LPeg. Sure, it might take a bit longer to generate code to parse some text, but it tends to be less “write only” than regular expressions.

Besides, you can do some pretty cool things with it. I have some LPeg code that will parse the following strftime() format string:

%A, %d %B %Y @ %H:%M:%S

and generate LPeg code that will parse:

Tuesday, 03 February 2015 @ 20:59:51

into:

date =
{
  min = 57.000000,
  wday = 4.000000,
  day = 4.000000,
  month = 2.000000,
  sec = 16.000000,
  hour = 20.000000,
  year = 2015.000000,
}

Or, if I set my locale correctly, I can turn this:

maŋŋebarga, 03 guovvamánu 2015 @ 21:00:21

into:

date =
{
  min = 0,000000,
  wday = 3,000000,
  day = 3,000000,
  month = 2,000000,
  sec = 21,000000,
  hour = 21,000000,
  year = 2015,000000,
}

But one annoyance that hits from time to time—named captures require a constant name. For instance, this pattern:

pattern = lpeg.Ct(
               lpeg.Cg(lpeg.P "A"^1,"class_a")
             * lpeg.P":" 
             * lpeg.Cg(lpeg.P "B"^1,"class_b") 
           )

(translated: when matching a string like AAAA:BBB, return a Lua table (lpeg.Ct()) with the As (lpeg.P()) in field class_a (lpeg.Cg()) and the Bs in field class_b)

applied to this string:

AAAA:BBB

returns this table:

{
  class_a = "AAAA",
  class_b = "BBB
}

The field names are constant—class_a and class_b. I'd like a field name based on the input. Now, there is a function lpeg.Cb() that is described as:

Creates a back capture. This pattern matches the empty string and produces the values produced by the most recent group capture named name.

Most recent means the last complete outermost group capture with the given name. A Complete capture means that the entire pattern corresponding to the capture has matched. An Outermost capture means that the capture is not inside another complete capture.

LPeg - Parsing Expression Grammars For Lua

A quick reading (and I'm guilty of this) leads me to think this:

pattern = lpeg.Cg(P"A"^1,"name")
        * lpeg.P":"
        * lpeg.Ct(lpeg.P "B"^1,lpeg.Cb("name"))

applied to the string:

AAAA:BBB

returns

{
  AAAA = "BBB"
}

But sadly, no. The only example of lpeg.Cb(), used to parse Lua long strings (which start with a “[”, zero or more “=”, another “[”, then text, ended with a “]”, zero or more “=” (but the number of “=” must equal the number of “=” between the two “[”) and a final “]”)):

equals = lpeg.P"="^0
open = "[" * lpeg.Cg(equals, "init") * "[" * lpeg.P"\n"^-1
close = "]" * lpeg.C(equals) * "]"
closeeq = lpeg.Cmt(close * lpeg.Cb("init"), function (s, i, a, b) return a == b end)
string = open * lpeg.C((lpeg.P(1) - closeeq)^0) * close / 1

shows that lpeg.Cb() was designed with this partular use case in mind—matching one pattern with the same pattern later on, and not what I want.

I can do what I want (a field name based upon the input) but the way to go about it is very klunky (in my opinion):

pattern = lpeg.Cf(              
                  lpeg.Ct("")    
                * lpeg.Cg(
                       lpeg.C(lpeg.P"A"^1)
                     * lpeg.P":"
                     * lpeg.C(lpeg.P"B"^1)
                   )
                ,function(acc,name,value)
                   acc[name] = value
                   return acc
                 end
        )

This is a “folding capture” (lpeg.Cf()) where we are accumulating our results (even though it's only one result—we have to do it this way) in a table (lpeg.Ct()) where each “value” is a group (lpeg.Cg()—the name is optional) consisting of a collection (lpeg.C() of As (lpeg.P()) followed by a colon (ignored), followed by a collection of Bs, all of which (except for the colon—remember, it's ignored) are passed to a function that assigns the string of Bs to a field name based on the string of As.

It gets even messier when you mix fixed field names with ones based upon the input. If all the field names are defined, it's easy to do something like:

eoln = P"\n"		-- match end of line
text = (P(1) - eoln)0	-- match anything but an end of line

pattern = lpeg.Ct(
		  P"field_one: "  * Cg(text^0,"field_one")   * eoln
		* P"field_two: "  * Cg(text^0,"field_two")   * eoln
		* P"field_three:" * Cg(text^0,"field_three") * eoln
)

against data like this:

field_one: Lorem ipsum dolor sit amet
field_two: consectetur adipiscing elit
field_three: Cras aliquet enim elit

to get this:

{
  field_one = "Lorem ipsum dolor sit amet",
  field_two = "consectetur adipiscing elit",
  field_three = "Cras aliquet enim elit"
}

But if we have some defined fields, but want to accept non-defined field names, then … well … yeah … I haven't found a nice way of doing it. And I find it annoying that I haven't found what I'm looking for.

Wednesday, February 04, 2015

A silly little file redirection trick under Unix

I'm in the process of writing a regression test for “Project: Sippy-Cup” and right now I'm more concentrating on writing what I call a “smoke-test”—something that can be run on my development machine after fixing bugs or adding features so that any obvious problems are “smoked out” before it hits the version control system.

Like “Project: Wolowizard,” this involves running multiple components. That isn't that much of an issue, I have plenty of Lua code to launch a program, and it typically looks like:

errno   = require "org.conman.errno"
syslog  = require "org.conman.syslog"
process = require "org.conman.process"

pid,err = process.fork()
if not pid then
  syslog('error',"fork() = %s",errno[err])
  os.exit(process.EXIT.SOFTWARE) -- who knew about /usr/include/sysexits.h?
elseif pid == 0 -- child process
  local stdin = io.open("/dev/null","r")
  local stdout = io.open("foobar.stdout.txt","w")
  local stderr = io.open("foobar.stderr.txt","w")

  -- --------------------------------------------------------------------
  -- redirect stdin, stdout and stderr to these files.  Once we've done
  -- the redirection, we can close the files---they're still "open" as
  -- stdin, stdout and stderr.  Then we attempt to start the program.  If
  -- that fails, there's not much we can do, so just exit the child
  -- process at that point.
  -- --------------------------------------------------------------------
  
  fsys.dup(stdin,fsys.STDIN)
  fsys.dup(stdout,fsys.STDOUT)
  fsys.dup(stderr,fsys.STDERR)

  stderr:close()
  stdout:close()
  stdin:close()

  process.exec(EXE,{ "–config" , "config.xml" })
  process.exit(process.EXIT.SOFTWARE)
end

Each program is launched in a similar manner, and if any of them crash, the testing harness gets notified. Also, once the tests are done, I can shutdown each process cleanly, all under program control. I want this to be a simple run-me type command that does everything.

During the testing of the testing program, it is nice to be able to see the output of the programs being tested. Sure, I have any output from the programs going to a file, but the problem with that is that it's hard to watch the output in real time. Upon startup (at least under Unix) if stdout (the normal output stream) is a terminal, the output appears a line at a time; otherwise, the output is “fully buffered”—meaning it's only actually written when there's around 4K or 8K worth of output, and if the programs aren't that chatty, you could be waiting a while if you're constantly checking the output files.

But there is a trick—this being Unix, you can redirect the output to another terminal (or in this modern age, a terminal window). I open up a spare terminal window (it's easy enough), and run the w command to find its device entry:

[spc]lucy:~>w
 20:31:32 up 15 days, 6 min,  3 users,  load average: 0.00, 0.00, 0.00
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
spc      pts/0    marvin.roswell.a 20:06   17.00s  0.28s  0.27s joe 2
spc      pts/1    marvin.roswell.a 20:15   15:50   0.03s  0.02s vi process.c
spc      pts/2    marvin.roswell.a 20:31    0.00s  0.01s  0.00s w
[spc]lucy:~>

Here, I can see that the w command is being run on terminal device /dev/pts/2 (under Linux, the “/dev/” portion isn't listed). So all I need to do is redirect stdout and stderr to /dev/pts/2 and the output will appear in that window, in real time.

So why do it in this roundabout way? Well, remember, I have several programs running. By opening up multiple terminal windows and directing the output of each program to these windows, the output from each program is kept separated and I can see what's going on. Then, when the testing program is working, I can then go back to writing the output to a file.

Oh, and under Mac OS-X:

spc]marvin:~>w
20:40  up 21 days,  1:20, 8 users, load averages: 0.01 0.05 0.08
USER     TTY      FROM              LOGIN@  IDLE WHAT
spc      console  -                14Jan15 21days -
spc      s000     -                18:48      32 -ssh XXXXXXXXXXXXXXXXXX
spc      s001     -                20:07      23 -bash
spc      s002     -                14Jan15 21days syslogintr
spc      s003     -                20:07       - w
spc      s004     -                20:07       - -ssh lucy
spc      s005     -                20:16       7 -ssh lucy
spc      s006     -                20:32       - -ssh lucy
[spc]marvin:~>

The “s003” now becomes /dev/ttys003.

---

Of course, statistics has little to say about Murphy's Law

Everyone knew it was coming. Second-and-1 on the 1-yard line. Marshawn Lynch was waiting in the backfield, poised to do what he was put on this Earth to do: Get a touchdown—this touchdown. The football gods had telegraphed how they wanted the game to end, directing a floating ball straight into Jermaine Kearse's hands. Beast Mode was going to drag the New England team kicking and screaming into the end zone if he had to. But the play call came in, Russell Wilson attempted a doomed pass that Malcolm Butler intercepted, and it was Seattle that punched and screamed its way off the field.

…

That's right. On the 1-yard line, QBs threw 66 touchdowns with no interceptions prior to Wilson's errant toss.³ Not mentioned: They also scored four touchdowns on scrambles (which Wilson is pretty good at last I checked). That's a 60.9 percent success rate.

Just for comparison's sake, here's how more than 200 runs fared this year in the same situation:

• 125 led to touchdowns.
• 94 failed to score.
• Of those, 23 were for loss of yardage.
• Two resulted in lost fumbles.

So overall, runs do a bit worse than passes (57.1 percent vs. 60.9 percent).

Via Robert Anstett on MyFaceGooglePlusSpaceBook, A Head Coach Botched The End Of The Super Bowl, And It Wasn’t Pete Carroll | FiveThirtyEight

I don't watch much football (if at all), but even I knew that last Seahawks play was not the right call. But actually, it may not have been the most idiotic thing for the Seahawks to do. The article goes deep into the math behind Pete Carroll's call.

Thursday, February 05, 2015

Bug hunt

Today T, a fellow cow-order here at The Ft. Lauderdale Office of The Corporation, found a particularly nasty bug in “Project: Sippy-Cup”—two identical SIP packets caused the request to fail. It uses UDP for the network transport, so duplicate packets are a possibility (so are dropped packets, but that's already being handled properly), something that the code didn't handle properly.

Now, “Project: Sippy-Cup” is written in Lua. I only say this because of how I handle requests. The main loop of the program waits for network packets using poll() (or epoll_wait() if running under Linux; they're like calls to select() (which is all you ever see in Unix networking tutorials because everybody is basing the code off examples written in the early 90s but I digress) only less CPU intensive). When poll() returns, I either create a coroutine or resume a coroutine (depending on where we are in the request) to handle the request.

I did it this way, even though I have to manually switch coroutines, to make the processing of requests easier to follow:

function handle(request)
  if not validate(request)
    socket:send(request.sender,"400 Bad client!  No cookie for you!")
    return
  end
  
  socket:send(request.sender,"200 Good boy!  You're such a good boy!")
  
  local info    = lookup_details(request)	-- blocking call
  local contact = dns.lookup(request.contact)	-- blocking call
  
  socket:send(contact,info)
  local reply = socket:recv()			-- blocking call
  
  if not reply.okay then
    log("I messed up")
  end
end

rather than having all this logic spread across multiple functions where each function represents a step in the processing:

-- not shown, the code that receives packets and calls the appropriate
-- routine.

function handle_incoming(state)
  if not validate(state.request)
    socket:send(state.request.sender,"400 Bad client!  No cookie for you!")
    return 
  end
  
  socket:send(state.request.sender,"200 Good boy!  You're such a good boy!")
  return lookup_details -- next function to call
end

function lookup_details(state)
  state.info = -- code to send request
  return contact_dns
end

function contact_dns(state)
  state.contact = state.request.contact
  -- code to send a DNS request
  return got_dns_response
end

function got_dns_response
  socket:send(state.contact,state.info)
  return process_reply
end

function process_reply(state)
  if not state.reply.okay then
    log("I messed up")
  end
end

It's fairly straightforward—once I get a SIP packet, I check a list to see if I've already created a coroutine based on the unique ID included in the packet. If one does exist, I then resume that coroutine; otherwise, I create one. The coroutine will run until it either finishes, or a function will block (say, a DNS query), and the code that could block will add the coroutine to a list it maintains and yield back to the main code.

The main coroutine, at 50,000 feet:

1. poll() returns.
2. It's a SIP packet: process it:
   1. Get the unique ID from the SIP packet.
   2. Use the ID to check the list of SIP coroutines. If a coroutine does not exist, create one and add it to the list.
   3. Resume the coroutine.
   4. If the coroutine has ended, remove the reference from the list of SIP coroutines; otherwise
   5. The coroutine has called coroutine.yield(), so keep the reference arond.
3. It's another type of packet (see below)
4. Go back to step 1.

Any routine that might block (and they're all network related) will put the coroutine on a list, and yield. So, when call dns.lookup(), that function does:

1. Do have have the data cached? If so, return it.
2. Otherwise, create the DNS request.
3. Send the DNS request to a DNS server.
4. Place the coroutine on a DNS list, indexed with the DNS request ID.
5. Call coroutine.yield(), which returns us to the main loop.
6. We resume here, so remove the coroutine from the DNS list.
7. Cache the data.
8. Return the data.

So, step 3 of the main coroutine contains these sub-steps:

1. Get the unique ID from the DNS packet.
2. Get the coroutine from the DNS list, indexed by the ID.
3. Resume that coroutine.

The lookup_detail() function (the handling of which would be step 4, which isn't shown) is similar to the DNS lookup. This all works and while the main driver code is a bit complex (and the complexity has to live somewhere), the actual processing code is nice and linear, making it easier to change the “business logic” in one place rather than have it scattered throughout the codebase.

Now we get to the bug—mishandling of duplicate packets. The first SIP packet comes in, has a unique ID, so a coroutine is created, we get to the call to lookup_details() and we yield, waiting for a reponse from our database engine (not a real database engine, but close enough). The duplicate SIP packet comes in. We already have a coroutine associated with this SIP ID, so re blindly resume it. The coroutine, freshly resumed, assumes the reply from the database has arrived and things go down hill from there (but only for that request; the program is still running and can handle new requests).

The solution is to realize we have a duplicate request and not resume the coroutine in that case.

And like most bugs, the fix is trivial—it's the hunt that's difficult.

Friday, February 06, 2015

The new video card, round three

The replacement replacement video card finally arrived! Woot!

The good: It's the right card!

The bad: It was not going into graphics mode when I run X.

The ugly: It took a few hours, but I finally got it working. It seems that when I tried the previous video card, the configuration was overwritten, and it took some gentle persuasion with a large hammer to straighten it out.

So now it's time to send back the wrong card, and relax now that this adventure is over.

---

Do not disturb the Happy Fun Sean

What I thought was going to be a very simple operation of “take the old video card out, put the new video card in and we're good to go” wasn't. And as typical of when one problem leads to another problem which leads to yet another problem, I get very focused while at the same time I get extremely angry. As the issues pile up, as I'm reading scores of web pages with conflicting information (that is, when they have an answer out of the hundreds of pages checked) and I'm building up this huge fragile structure in my mind of everything I've tried, what I've yet to try, and futher questions to answer—

“What would you like for lunch?”

And it collapses like a house of cards, leaving me to start all over again.

I'll admit, it can be hard to distinguish between this and “very focused while working in a program” just prior to the house of cards collapse, but my response in the former is a primal scream while the my reponse to the later requires scraping me off the ceiling.

Bunny has learned to live with it.

---

The Alton Brown Edible and Inevitable Tour

Yeast.

The preshow consists of sock puppets of belching yeast.

And farts. Sock puppets of belching and farting yeast.

In time to the Alfred Hitchcock Theme.

Of course it can only mean one thing—Bunny and I had tickets to the Alton Brown Edible Inevitable Tour!

Alton was nice enough to take our picture and post it to Twitter.

My god—what a show!

Alton Brown is extremely funny. After an attempt at rapping (he should not quit his day job, that's all I have to say about that), he went into “Things He Could Not Say On Televsion” which include rants against chicken fingers (“when I am depressed, or sad, or even bored, I just have to remember the sound of little girls screaming … ”) to trout ice cream.

He makes a gallon of chocolate ice cream in 10 seconds (and it does not involve liquid nitrogen) and using the worlds largest Easy•Bake Oven (cranked to 11, it can be seen from outer space) he cooks pizza in four minutes.

And I best not forget his country song to Airline Lounge Shimp Cocktail. Rapping aside, he's not that bad of a singer and I've certainly heard worse.

All I can say is, if you are a fan of Good Eats or of Alton Brown in general, you should go. It's easily worth the ticket price.

And yes, he always carries nutmeg in his pocket.

Saturday, February 07, 2015

The VC model to R&D

There's a fascinating article about Xerox PARC (link via Hacker News) and how Xerox dropped the ball on computing (only to have Apple pick up the ball and run with it, followed closely by Microsoft).

But what most people don't realize is that Xerox was never in the computer business, and I think licensing the technology to Apple was probably a smart move. But to me, this is the key quote:

“Xerox did research outside their business model, and when you do that you should not be surprised that you have a hard time dealing with it—any more than if some bright guy at Pfizer wrote a word processor. Good luck to Pfizer getting into the word-processing business. Meanwhile, the thing that they invented that was similar to their own business—a really big machine that spit paper out—they made a lot of money on it.” And so they did. Gary Starkweather’s laser printer made billions for Xerox. It paid for every other single project at Xerox PARC, many times over.

Creation Myth - The New Yorker

That quote, “[i]t paid for every other single project at Xerox PARC, many times over,” is, to me, the key quote as to why large companies should always invest in research and development. Sure, not every idea will pan out, but those that do pan out, really pan out! (Heck, venture capitalists do that today—fund a bunch of projects, most of which will fail, but the ones that don't make tons of money)

Sunday, February 08, 2015

As if a ½K Chess program wasn't hardcore enough

These next few posts are a technical write-up of how I created 8088 Domination, which is a program that displays fairly decent full-motion video on a 1981 IBM PC.

Via Hacker News, 8088 Domination Post-Mortem, Part 1 « Oldskooler Ramblings

Yes, a 4.77MHz 8088 based computer with an effective harddrive transfer rate of 90KB/s can do full motion video.

Okay, it's not a great video image, but still, it's an impressive display of coding expertise and thinking outside of the box.

Monday, February 09, 2015

I just kind of wish Lua could catch these types of bugs automatically like a C compiler can

Another day, another bug in “Project: Sippy-Cup.” This time a simple typo, the type of typo that a compiler would have caught before the code was checked in. And it's the type of bug that is the most prevalent in my code.

I'm beginning to see why unit testing is so popular with the scripting/dynamic languages—they catch the types of bugs that compilers of static languages already catch.

On the down side, unit testing ensures that I get to play “compiler” far more often than I care to …

Sigh.

Tuesday, February 10, 2015

I wonder if anyone has read 100 Years of Solitude 100 times?

I have read two books more than a 100 times, for different motives and with different consequences. Hamlet I read a 100 times for my dissertation, The Inimitable Jeeves by PG Wodehouse a 100 times for comfort. The experience is distinct from all other kinds of reading. I’m calling it centireading.

I read Hamlet a 100 times because of Anthony Hopkins. He once mentioned, in an interview with Backstage magazine, that he typically reads his scripts over a 100 times, which gives him “a tremendous sense of ease and the power of confidence” over the material. I was writing a good chunk of my doctoral dissertation on Hamlet and I needed all the sense of ease and power of confidence I could muster.

…

It’s not necessarily the quality. The Inimitable Jeeves does not contain the best Wodehouse story. That is either Lord Emsworth and the Girl Friend, which Rudyard Kipling called “the perfect short story” or Uncle Fred Flits By collected in Young Men in Spats. But there are no dull moments in The Inimitable Jeeves, no bad parts. Each plot is a novelty, without a trace of laziness. There is not a single weak verb in the entire book.

Via Hacker News, Centireading force: why reading a book 100 times is a great idea | Books | The Guardian

Interesting. I wonder what my friend Hoade would have to say about this?

For me, I don't think I've read any book a hundred times, although I might be in the upper two digits for a few books.

One book I've read so many times my copy is falling apart is Hackers: Heros of the Computer Revolution. It's about three groups of iconic programmers, the MIT graduate students in the 60s (Green blatt, Gosper, Tom Knight, etc.); the Homebrew Computer Club group from the 70s (Woz, Felsenstein, Captain Crunch, etc.), and the scores of teenagers banging out computer video games in the 80s (John Harris—all of this is from memory and I'll stop now before I post the entire book from memory). I can certainly say it had an effect on me and heavily influenced my views on programming.

Another book I've read dozens of times is Have Space Suit, Will Travel, the first science fiction book I read (so it's heavy with nostalgia), and one that doomed me to reading science fiction almost exclusively. My favorite part of the book is when Kip (the main protagonist) is imprisoned on Pluto and he spends his time figuring out how fast he travelled to Pluto by knowing both the time (five days) and distance (30 AU), while upset that he doesn't have his slipstick to help with the calculations (and I now have a few slipsticks of my own because of this book).

I'm not sure what that says about me, though.

Another book I've read uncountable times is Snow Crash, a wild science-fiction cyberpunk romp of the near future where the United States Government is just another franchise, and the main villain is a cross between H. Ross Perot and L. Ron Hubbard whose hired hand, an Aleut harpoonist, has been declared a sovereign nation unto himself (for he has his own tactical nuclear weapon).

Yes, it sounds insane describing it, but it is really good, really funny, and again, I'm not sure what it says about me that my favorite bits are the exposition-heavy bits about neurolinguistic hacking.

There aren't many other books I've read quite as often as those, but if you include comic books, then there are about a dozen or so Uncle Scrooge comic books I probably have read a hundred times or so …

Wednesday, February 11, 2015

So if we “do it like they do on the Discovery Channel,” does that mean they “do it like we do on Reefer Madness?”

Noah’s Ark, he found, would have looked a lot like London on a Saturday night. “In every country, in almost every class of animal,” Siegel explains, “I found examples of not only the accidental but the intentional use of drugs.” In West Bengal, a group of 150 elephants smashed their way into a warehouse and drank a massive amount of moonshine. They got so drunk they went on a rampage and killed five people, as well as demolishing seven concrete buildings. If you give hash to male mice, they become horny and seek out females — but then they find “they can barely crawl over the females, let alone mount them,” so after a little while they yawn and start licking their own penises.

In Vietnam, the water buffalo have always shunned the local opium plants. They don’t like them. But when the American bombs started to fall all around them during the war, the buffalo left their normal grazing grounds, broke into the opium fields, and began to chew. They would then look a little dizzy and dulled. When they were traumatized, it seems, they wanted — like the mongoose, like us — to escape from their thoughts.

Via Hacker News, Why animals eat psychoactive plants - Boing Boing

Posted mostly for a few friends who shall remain nameless who might find this article interesting.

You know who you are.

Thursday, February 12, 2015

Sixteen hours to change three lines of code

The bug report was simple enough: “Project: Sippy-Cup” would, when it had to retransmit the data (remember, we're running over UDP), it would send an additional packet.

Data flow for “Project: Sippy-Cup”

Sippy Cup	direction	Client
DATA	→	(packet dropped)
DATA	→	(packet received)
(packet recieved)	←	ACK
DATA	→

So far, the bugs found in “Project: Sippy-Cup” have been easy to reproduce (thankfully!) and easy to fix once identified. This one was easy to reproduce … and took two days of intense effort to fix. The bug is not a show-stopper, and the client should (there's that word) handle the duplicate packet. My manager, S, was subtly hinting that I should put this on the backburner, but this is one of those problems that just irks me.

I added logging statements. I instrumented the code. I traced the execution of the program line-by-line (easy enough to do within Lua as there's a pre-defined method for hooking into the execution engine, and even though this didn't reveal the root cause of the bug, it did reveal a few places where some simple changes optimized the code (but that's for another post). For example, here's all the code that's executed when you run the luarocks command, but fair warning—it's nearly 7,000 lines long) and it became clear that something was wrong with the timeout handling (what I didn't mention the other day about scheduling coroutines is how I handle timeouts, but alas, that's for another post as well). The code is basically:

request = receive_packet_from_client()
send_ack()
send_request_to_db(request)

-- ------------------------------------------------------------------
-- This sets things up to force the coroutine to resume in one second,
-- returning a result of nil.
-- ------------------------------------------------------------------

timeout(1)

-- ----------------------------------------------------------------
-- the code that resumes this coroutine will pass in the result in
-- coroutine.resume(), which is assigned to dbresult
-- ----------------------------------------------------------------

dbresult = coroutine.yield()

-- ------------------------------------------------------------------------
-- cancel a pending timeout, if any.  If we did timeout, this is a NOP.  If
-- we haven't timed out, this prevents the timeout from triggering.
-- ------------------------------------------------------------------------

timeout(0)

if not dbresult then
  info = failed()
else
  info = extract(dbresult)
end

for retries = 1 , 3 do
  send_information_to_client()
  timeout(1)
  ack = coroutine.yield()
  timeout(0)
  if ack then
    return
  end
end

I was able to determine that the initial “timeout” was the request to the database engine timeout trigger (after hacking in a way to track the timeout requests) which meant the timeout code wasn't working properly. Once that was clear, I was able to locate the problem code:

-- ------------------------------------------------------------------------
-- TQUEUE contains references to coroutines that want a timeout.  If we set
-- the timeout to 0, this just sets the awake field to 0, which informs the
-- coroutine scheduler not to run the coroutine when it "times out".  
-- ------------------------------------------------------------------------

function timeout(when)
  local f = coroutine.running()

  if when == 0 then
    if TQUEUE[co] then
      TQUEUE[co].awake = 0 -- oops, bug.  See this post for details
    end
  else
    TQUEUE:insert(when,f)
  end
end

Here we have an instance where a static compiler would have flagged this immediately. I either need to rename f to co, or change co to f. A simple two line fix (and I really need to find a code checker for Lua).

The other issue, now quickly found, was this bit of code:

local function handler(info,err,remote,message,startime)
  ... 
  local ninfo,err,nremote,nmessage = coroutine.yield()
  timeout(0)

  if err == 0 then
    if not message.request.status then
      -- error
    else
      return nmessage
    end
  else 
  ...

Even a static compiler would not have found this issue—I was checking the wrong variable! The code should read:

if not nmessage.request.status then

Bad me for using a bad name for a variable. No cookie for me!

Okay, this is a simple one line change. Total lines changed: 3.

Friday, February 13, 2015

A missed optimization in Lua

Yesterday, I made brief mention of optimizing some Lua code, and said it was for another post.

This is said post.

The code in question (not identical, but this exhibits the same problem):

             require "ddt".tron() -- trace the execution
local lpeg = require "lpeg"

local Cs = lpeg.Cs
local C  = lpeg.C
local R  = lpeg.R
local P  = lpeg.P

function canonical(header)
  local function Pi(text)
  
    local pattern = P(true)
    for c in text:gmatch(".") do
      pattern = pattern * (P(c:lower()) + P(c:upper()))
    end
    
    return pattern / text
  end
  
  local ALPHA = R("AZ","az")
  
  local id = Pi "ID" -- exceptions to Camel-Case
           + Pi "MIME"
           + Pi "CSeq"
  
  local word = (C(ALPHA) * C(ALPHA^0))
             / function(i,r)
                 return i:upper() .. r:lower()
               end
  local other = C((P(1) - ALPHA)^1)
  local hdr   = Cs((id + word + other)^1)
  
  return hdr:match(header)
end

print(canonical "return-from")
print(canonical "message-id")
print(canonical "mime-version")
print(canonical "cseq")

The code in question transforms a header name like return-from to the canonical form Return-From; it'll also transform ReTuRn-FRom into the canonical form. The code is used to match headers in Internet based messages like email, HTTP or SIP (as the header names need to match case-insensitively—I'll leave how it works to you, the reader (here are some hints) since what the code does isn't germane to this discussion). Now, when you trace the execution, you'll notice something:

@./ddt.lua             187: end
@code.lua                2: local lpeg = require "lpeg"
@code.lua                4: local Cs = lpeg.Cs
@code.lua                5: local C  = lpeg.C
@code.lua                6: local R  = lpeg.R
@code.lua                7: local P  = lpeg.P
@code.lua               34: end
@code.lua                9: function canonical(header)
@code.lua               36: print(canonical "return-from")
@code.lua               18:   end
@code.lua               20:   local ALPHA = R("AZ","az")
@code.lua               22:   local id = Pi "ID" -- exceptions to Camel-Case
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               23:            + Pi "MIME"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               24:            + Pi "CSeq"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               26:   local word = (C(ALPHA) * C(ALPHA^0))
@code.lua               29:                end
@code.lua               30:   local other = C((P(1) - ALPHA)^1)
@code.lua               31:   local hdr   = Cs((id + word + other)^1)
@code.lua               33:   return hdr:match(header)
@code.lua               28:                  return i:upper() .. r:lower()
@code.lua               28:                  return i:upper() .. r:lower()
@code.lua               37: print(canonical "message-id")
@code.lua               18:   end
@code.lua               20:   local ALPHA = R("AZ","az")
@code.lua               22:   local id = Pi "ID" -- exceptions to Camel-Case
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               23:            + Pi "MIME"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               24:            + Pi "CSeq"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               26:   local word = (C(ALPHA) * C(ALPHA^0))
@code.lua               29:                end
@code.lua               30:   local other = C((P(1) - ALPHA)^1)
@code.lua               31:   local hdr   = Cs((id + word + other)^1)
@code.lua               33:   return hdr:match(header)
@code.lua               28:                  return i:upper() .. r:lower()
@code.lua               38: print(canonical "mime-version")
@code.lua               18:   end
@code.lua               20:   local ALPHA = R("AZ","az")
@code.lua               22:   local id = Pi "ID" -- exceptions to Camel-Case
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               23:            + Pi "MIME"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               24:            + Pi "CSeq"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               26:   local word = (C(ALPHA) * C(ALPHA^0))
@code.lua               29:                end
@code.lua               30:   local other = C((P(1) - ALPHA)^1)
@code.lua               31:   local hdr   = Cs((id + word + other)^1)
@code.lua               33:   return hdr:match(header)
@code.lua               28:                  return i:upper() .. r:lower()
@code.lua               39: print(canonical "cseq")
@code.lua               18:   end
@code.lua               20:   local ALPHA = R("AZ","az")
@code.lua               22:   local id = Pi "ID" -- exceptions to Camel-Case
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               23:            + Pi "MIME"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               24:            + Pi "CSeq"
@code.lua               12:     local pattern = P(true)
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@code.lua               13:     for c in text:gmatch(".") do
@code.lua               17:     return pattern / text
@code.lua               26:   local word = (C(ALPHA) * C(ALPHA^0))
@code.lua               29:                end
@code.lua               30:   local other = C((P(1) - ALPHA)^1)
@code.lua               31:   local hdr   = Cs((id + word + other)^1)
@code.lua               33:   return hdr:match(header)

There's quite a bit of code executed. That's because the Lua compiler isn't sufficiently smart to notice that most of the code in canonical() never changes—it's independent of the passed in parameter and thus, it could be compiled once. And it's this behavior that I noticed the other day. It's an easy fix (just lift the invarient code out of the function body) and the results are about a third the processing:

@./ddt.lua             187: end
@c3.lua                  2: local lpeg = require "lpeg"
@c3.lua                  4: local Cs = lpeg.Cs
@c3.lua                  5: local C  = lpeg.C
@c3.lua                  6: local R  = lpeg.R
@c3.lua                  7: local P  = lpeg.P
@c3.lua                 18:   end
@c3.lua                 20:   local ALPHA = R("AZ","az")
@c3.lua                 22:   local id = Pi "ID" -- exceptions to Camel-Case
@c3.lua                 12:     local pattern = P(true)
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 17:     return pattern / text
@c3.lua                 23:            + Pi "MIME"
@c3.lua                 12:     local pattern = P(true)
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 17:     return pattern / text
@c3.lua                 24:            + Pi "CSeq"
@c3.lua                 12:     local pattern = P(true)
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 14:       pattern = pattern * (P(c:lower()) + P(c:upper()))
@c3.lua                 13:     for c in text:gmatch(".") do
@c3.lua                 17:     return pattern / text
@c3.lua                 26:   local word = (C(ALPHA) * C(ALPHA^0))
@c3.lua                 29:                end
@c3.lua                 30:   local other = C((P(1) - ALPHA)^1)
@c3.lua                 31:   local hdr   = Cs((id + word + other)^1)
@c3.lua                 35:   end
@c3.lua                 33:   function canonical(header)
@c3.lua                 35:   end
@c3.lua                 38: print(canonical "return-from")
@c3.lua                 34:   return hdr:match(header)
@c3.lua                 28:                  return i:upper() .. r:lower()
@c3.lua                 28:                  return i:upper() .. r:lower()
@c3.lua                 39: print(canonical "message-id")
@c3.lua                 34:   return hdr:match(header)
@c3.lua                 28:                  return i:upper() .. r:lower()
@c3.lua                 40: print(canonical "mime-version")
@c3.lua                 34:   return hdr:match(header)
@c3.lua                 28:                  return i:upper() .. r:lower()
@c3.lua                 41: print(canonical "cseq")
@c3.lua                 34:   return hdr:match(header)

I also recorded the execution with LuaJIT (it's faster because it compiles Lua into native code) and it, too, is not sufficiently smart to lift the constant code out of the function. It may be that detecting this is hard for a compiler to do, or that such transformations might not be considered safe (due to possible side effects).

In any case, I found it a bit surprising (although in retrospect, it shouldn't have been).

Saturday, February 14, 2015

I do belieive it's a holiday today

Sunday, February 15, 2015

Notes on an overheard conversation about a bouquet of flowers bought for Saint Valentine's Day

“Did you see what I did to the flowers you gave me?”

“…”

“…”

“…”

“…”

“Oh! You added some flowers to it!”

“I was starting to get worried there for a moment, but you managed to save yourself.”

Monday, February 16, 2015

Timing out Lua coroutines

Time for “that's for another post”—this time, handling timeouts in Lua coroutines.

So, I have this Lua coroutine running, and I want to make a DNS request:

host = dns.request("www.conman.org.","a")

It takes a non-trivial amount of time—time that could be used to run other coroutines. But the operation could take longer than expected (say, two seconds, one second longer than we want to wait) or never even complete (because we lost the request packet, or the reply packet). In that case, we want to timeout the operation.

Step one—let's set a limit to how long we can run:

timeout(1)                                    -- time out in one second
host,err = dns.request("www.conman.org.","a") -- DNS query
timeout(0)                                    --  no longer need the timeout

The trick now is to implement the code behind timeout(). First we need to have some way of storing the coroutines (technically, a reference to the coroutine) that are waiting to timeout, and some easy way to determine which ones are timed out. For this, I used a binary heap, or techically, a “binary min heap” to store the coroutine reference. The “min” variation because the nodes are ordered by the “awake” time, and times in the future are larger (and thus, the next coroutine that will timeout next will always appear in the first spot in the binary min heap).

Along with the awake value and the coroutine reference, I have a trigger flag. This flag is important for the “cancelling a timeout” case. An earlier version of the code searched sequentially through the list, making “cancelling” an expensive operation (O(n)) compared to “setting” a timeout (O(log(n))). I decided it was easier to just have a flag, and keep a secondary index, keyed by coroutine, to the correct node in the binary min tree. That way, cancelling a timeout is cheap (an amortized O(1)) with the following code:

function timeout(when)
  local co = coroutine.running()

  if when == 0 then
    if TQUEUE[co] then           -- guard to make sure we have an entry
      TQUEUE[co].trigger = false -- disarm timeout trigger
    end
  else
    TQUEUE:insert(when,co)       -- otherwise, set the timeout
  end
end

(I should note that the code I presented last Thursday was buggy—I realized I wasn't keeping the invariant condition necessary for a binary min heap (child nodes have a larger value than the parent node) by setting the awake field to 0 (a child would then have a smaller value than its parent)—it didn't break the code but it did make it behave a bit oddly)

I also maintain a run queue—a list of coroutines that are ready to run, used in the main loop:

function mainloop()
  local timeout = -1            -- the timeout to wait for new packets
  local now     = clock.get()

  while #TQUEUE > 0 do          -- check timeout queue
    local obj = TQUEUE[1]       -- next coroutine to timeout
    
    if obj.trigger then         -- do we trigger?
      timeout = obj.awake - now -- if so, when
      if timeout > 0 then       -- if now now (or before) 
        break                   -- stop looking through the timeout queue
      end
      
      table.insert(RQUEUE,obj)  -- insert coroutine into the run queue
    end
    
    TQUEUE:delete()             -- delete the entry from the timeout queue
  end
  
  if #RQUEUE > 0 then           -- if there's anything in the run queue
    timeout = 0                 -- we set our timeout
  end
  
  -- ---------------------------------------------------
  -- Receive packets and process---see this post for some details; the actual
  -- code this isn't
  -- ---------------------------------------------------

  for _,event in ipairs(poll:events(timeout)) do
    event.obj(event)
  end
  
  -- -----------------------------------------------------------
  -- run each coroutine in the run queue (some details snipped)
  -- -----------------------------------------------------------
  
  while #RQUEUE > 0 do
    local obj = table.remove(RQUEUE,1)
    if coroutine.status(obj.co) == 'suspended' then
      coroutine.resume(obj.co)
    end
  end
  
  return mainloop()
end

The code initially sets the timeout to wait for network activity to -1, or “wait indefinitely.” It then runs through the timeout queue, checking each item to see if there's a coroutine that's triggered to run. If it comes across a coroutine that has been “disarmed” then it's simply ignored and deleted from the timeout queue. Otherwise, it checks to see if the awake time is still in the future and if so, it's done with the timeout queue; otherwise, it moves the coroutine reference to the run queue and deletes it from the timeout queue.

Eventually, it'll get out of tha loop with a timeout of “indefinite” (if there were no coroutines waiting for a timeout) or some fixed amount of time until the next coroutine times out. It then checks the run queue, and if there is anything in that list, it sets the network activity timeout to 0 (if there is no packets, return immedately).

It then checks for network packets and for each one, process the packet (which in turn might schedule more coroutines to run). After that, it then goes through the run queue, resuming each coroutine in the list. After that, it starts over again.

Tuesday, February 17, 2015

It's a limitation in implementation, not in specification

(I'm responding inline, and I'm not looking at all the aforementioned posts/comments, so forgive me if I'm missing something here.)

It sounds like you're arguing that SRV records are no slower than A records, which, on it's face, seems reasonable. A DNS request is a DNS request, and aside from a response being too big for UDP and having to switch to TCP, you should get nearly-identical performance.

The part, to me, that looks like a real performance issue is potentially having to double the minimum amount of queries to serve a website. We couldn't possibly switch directly to SRV records; there would have to be an overlap of browsers using both SRV and A records for backwards compatibility.

If we stick with that invariant, then we can say that the first-load cost of a page not using SRV records doubles in the worst case: websites that only have an A record. Now we're looking for a SRV record, not getting it, and falling back to an A record. So, now, all of the normal websites who don't give a crap about SRV records, won't ever use them, pay a performance penalty. A marginal one, sure. but it's there.,

So, overall, their claim seems valid, even if low in severity.

I'd love to hear from someone that has the data, but I can count on one hand the number of times where a loss of IP connectivity has happened where I wish I had SRV records for load balancing. It's usually bad DNS records, or slow/bad DNS propagation, or web servers behind my load balancers went down, or a ton of other things. Your point is still totally valid about being able to more easily load balance across multiple providers, datacenters, what have you… but I'm not convinced it's as much of a problem as you make it out to be.

Hacker News comment about DNS

The argument being presented is that DNS SRV resource records should be used more often than they are (I agree), but that to use them for HTTP during a transitional period would lead to a slower web browsing experience as two (or more) DNS lookups would be required— one for the SRV RR, and if that failed, a followup A RR lookup for the IP address (or worse, an AAAA RR lookup for an IPv6 address and then fallback to an A RR lookup). But that's only the case because of a stupid limitation on a particular implementation of DNS that happens to hold, oh, 80% of the DNS traffic on the web (and 97% of all statistics are made up on the spot).

I only noticed this when I wrote my own DNS parsing routines and was testing them out. The specification, RFC- 1035, describes the format for a DNS packet:

DNS Packet Layout

Header
Question	what RRs are we interested in
Answer	RRs that match our question
Authority	RRs pointing toward an authority
Addtional	RRs holding additional information

And the “Header” section contains four 16 bit fields, described as such:

QDCOUNT         an unsigned 16 bit integer specifying the number of
                entries in the question section.

ANCOUNT         an unsigned 16 bit integer specifying the number of
                resource records in the answer section.

NSCOUNT         an unsigned 16 bit integer specifying the number of name
                server resource records in the authority records
                section.

ARCOUNT         an unsigned 16 bit integer specifying the number of
                resource records in the additional records section.

RFC-1035: Domain Implementation and Specification"

Take special note of the QDCOUNT field—“number of entries in the question section.”

Nowhere in RFC-1035 states that “QDCOUNT shall be one, and one shall be the only number in QDCOUNT, which shall be one, and nothing more shall appear therein. Amen.” Nowhere.

So, in theory, a program can ask for the following RRs in a single DNS packet: SRV, AAAA and A. But sadly, in theory, there is no difference between theory and practice, but in practice … sigh. You need three separate queries because the most popular DNS server in use decided to support only a single question.

Wednesday, February 18, 2015

Remains of the cake

Sadly for Edvard, someone got to the cake before he did.

---

A limitation sure, but it sneaks around its own limitations

I was reminded by Yesterday's post on DNS that I was playing around with SRV records, and oh yes, here's a few I set up some time ago:

_http._tcp              IN      SRV     1  1 8888 lucy
                        IN      SRV     2 10 8888 marvin
                        IN      SRV     2 20 8888 bunny-desktop
                        IN      SRV     2 30 8888 saltmine

So, let's see what I get when I query _http._tcp.roswell.area51. (the “home domain” I use around here):

  rcode = 0.000000,
  ra = true,
  aa = true,
  query = false,
  ad = false,
  rd = true,
  cd = false,
  tc = false,
  id = 1234.000000,

  question =
  {
    class = "IN",
    name = "_http._tcp.roswell.area51.",
    type = "SRV",
  },

Okay, so far, this is expected—some DNS flags, and the “question” we asked. Continuing …

  answers =
  {
    [1] =
    {
      port = 8888.000000,
      type = "SRV",
      name = "_http._tcp.roswell.area51.",
      weight = 30.000000,
      target = "saltmine.roswell.area51.",
      class = "IN",
      ttl = 86400.000000,
      priority = 2.000000,
    },
    [2] =
    {
      port = 8888.000000,
      type = "SRV",
      name = "_http._tcp.roswell.area51.",
      weight = 1.000000,
      target = "lucy.roswell.area51.",
      class = "IN",
      ttl = 86400.000000,
      priority = 1.000000,
    },
    [3] =
    {
      port = 8888.000000,
      type = "SRV",
      name = "_http._tcp.roswell.area51.",
      weight = 10.000000,
      target = "marvin.roswell.area51.",
      class = "IN",
      ttl = 86400.000000,
      priority = 2.000000,
    },
    [4] =
    {
      port = 8888.000000,
      type = "SRV",
      name = "_http._tcp.roswell.area51.",
      weight = 20.000000,
      target = "bunny-desktop.roswell.area51.",
      class = "IN",
      ttl = 86400.000000,
      priority = 2.000000,
    },
  },

Okay, these are the answers we were looking for, but wait? What's this?

  additional =
  {
    [1] =
    {
      type = "A",
      name = "lucy.roswell.area51.",
      address = "192.168.1.10",
      class = "IN",
      ttl = 86400.000000,
    },
    [2] =
    {
      type = "AAAA",
      name = "lucy.roswell.area51.",
      address = "fc00::1",
      class = "IN",
      ttl = 86400.000000,
    },
    [3] =
    {
      type = "A",
      name = "marvin.roswell.area51.",
      address = "192.168.1.13",
      class = "IN",
      ttl = 86400.000000,
    },
    [4] =
    {
      type = "AAAA",
      name = "marvin.roswell.area51.",
      address = "fc00::3",
      class = "IN",
      ttl = 86400.000000,
    },
    [5] =
    {
      type = "A",
      name = "bunny-desktop.roswell.area51.",
      address = "192.168.1.16",
      class = "IN",
      ttl = 86400.000000,
    },
    [6] =
    {
      type = "AAAA",
      name = "bunny-desktop.roswell.area51.",
      address = "fc00::2",
      class = "IN",
      ttl = 86400.000000,
    },
    [7] =
    {
      type = "A",
      name = "saltmine.roswell.area51.",
      address = "192.168.1.18",
      class = "IN",
      ttl = 86400.000000,
    },
    [8] =
    {
      type = "AAAA",
      name = "saltmine.roswell.area51.",
      address = "fc00::4",
      class = "IN",
      ttl = 86400.000000,
    },
  },

Yes, even though I run that braindead DNS implementation that only accepts single questions, it also preemptively sent back the appropriate IP addresses!

Will wonders never cease. There's probably no need for multiple DNS queries to handle SRV lookups.

Thursday, February 19, 2015

“Some people, when confronted with a problem, think ‘I know, I'll use regular expressions.’ Now they have two problems.”

All I can say is that I'm very happy that I'm writing “Project: Sippy-Cup” in Lua. Lua not only makes text manipulation easy, (heck, most scripting languages like Python or Ruby can easily manipulate text) but also that it has the wonderful LPeg that makes writing parsers not only easy, but easier to understand, unlike regular expressions.

And processing SIP messages is pretty much nothing but text parsing and manipulation, with a little bit of networking on the side.

Friday, February 20, 2015

Requiem for a dream

One day while sitting on the couch I noticed that the perspective of the lamp was odd, like inverted. It was still in 3D but… just‥ wrong. (It was a square lamp base, red with gold trim on 4 legs and a white square shade). I was transfixed, I couldn't look away from it. I stayed up all night staring at it, the next morning I didn't go to work, something was just not right about that lamp.

I stopped eating, I left the couch only to use the bathroom at first, soon I stopped that too as I wasn't eating or drinking. I stared at the fucking lamp for 3 days before my wife got really worried, she had someone come and try to talk to me, by this time my cognizance was breaking up and my wife was freaking out. She took the kids to her mother's house just before I had my epiphany…. the lamp is not real…. the house is not real, my wife, my kids… none of that is real… the last 10 years of my life are not fucking real!

Via Jason Kottke, temptotosssoon comments on Have you ever felt a deep personal connection to a person you met in a dream only to wake up feeling terrible because you realize they never existed?

There's a Star Trek: The Next Generation episode, “The Inner Light,” where Picard is knocked unconscience by an alien probe for about half an hour, but Picard spends a subjective thirty years living life on an alient planet. It's considered one of the best Star Trek episodes and won a ton of awards.

The article presented above appears to be a real-life occurrence of subjectively living years of your life in a dream-like state. Only without the alien probes. And the flute.

Saturday, February 21, 2015

Procedures in Experimental Physics

I never epxected that mentioning the book Procedures in Experimental Physics on Hacker News would prove to be so popular.

Wow.

---

The twilight of quantum mechanics

Quantum mechanics is weird only because we don't learn statistics in high school (well I didn't anyway), and we can't come up with good real- life analogies for quantum interactions.

For example, the twin-slit experiment used to illustrate collapsing the wave function (a single electron fired through 2 slits will show a wave interference pattern on the wall, but the pattern disappears if you find out which slit the electron passed through) is portrayed by physicists as obscure, weird, arcane, or even as indecipherable devil magic which us mere mortals can never strive to intuitively understand beyond pulling out a PDE.

This flat-out isn't true, and here is my analogy for the 2x slit experiment in real life (using trashy fiction):

The electron is an young impressionable female, slit A is the handsome vampire, and slit B is the wild werewolf. Until absolutely forced to pick one of the slits, the electron sort of strings both slits along (and the result is a lot of interference which, in the literary world, we call plot). But, when the reader looks at the end, she (the electron) inevitable picks one of the slits. Summed over all the trashy romance fiction out there, one gets the feeling it's the same damn electron and two slits everywhere, yet she is clearly making different decisions each time.

Quantum mechanics is weird only because we don't learn statistics in high school… | Hacker News

I really have nothing else to add.

Sunday, February 22, 2015

Adult Wednesday Addams

Oooh! Cool! The new season of Adult Wednesday Addams is here!

This web series is based more on the severe, deadpan witted Wednesday Addams of the movie rather than sweet-natured but macabre Wednesday Addams of the TV show, which makes it worth watching (in my opinion).

Monday, February 23, 2015

Looking Glass Falls is looking a bit brittle these days

On the way back from Hike From Hell (an appropriate name, given we hiked to the Devil's Courthouse) we did stop off at Looking Glass Falls:

But that was back in October.

Earlier this week, it wasn't quite so moving, what with the weather in Brevard, NC being quite a bit below freezing.

You know, I never gave it any thought if waterfalls could freeze …

Tuesday, February 24, 2015

Notes from lunch at a Thai restaurant somewhere in Ft. Lauderdale, Florida

As I was sliding my way into the booth, I noticed that the older gentleman in the booth next to ours had an uncanny resemblance to William Gibson, a writer well known for his cyberpunk novels Neuromancer, Count Zero and Mona Lisa Overdrive. It was amazing to think that I might be in the presense of the William Gibson, but I did a quick scan of the restaurant, on the lookout for Neal Stephenson, cyberpunk author as well, known for Snow Crash, The Diamond Age and Cryptonomicon. It was too much to hope for, and too terrible to contemplate the both of them being in the same place.

---

If you play in someone else's sandbox, don't get upset as they change the rules on you—it's not your sandbox to do as you please

While I realize that I have the knowledge and skills to run my own blog on my own server, I am, nonetheless, grateful that I can, for it appears that Google is now dictating what you can and cannot post publicly on Blogger (link via Flutterby).

Wednesday, February 25, 2015

The unintentional conspiracy

So why am I typing this on a laptop running GNU/Linux, the free software operating system, not an Apple or Windows machine? And why are my phones and tablets running a privacy-enhanced offshoot of Android called Cyanogenmod, not Apple’s iOS or standard Android?

Because, first of all, I can get my work done fine. I can play games. I can surf endlessly. The platform alternatives have reached a stage where they’re capable of handling just about everything I need.

…

Control is moving back to the center, where powerful companies and governments are creating choke points. They are using those choke points to destroy our privacy, limit our freedom of expression, and lock down culture and commerce. Too often, we give them our permission—trading liberty for convenience—but a lot of this is being done without our knowledge, much less permission.

Via Reddit, Why I’m Saying Goodbye to Apple, Google and Microsoft — Backchannel — Medium

While I'm sympathetic to his views, I don't believe there's one vast conspiracy to restrict consumers' use of computers. Each step, taken in isolation, is understandable, but when viewed over time, can appear to be a vast conspiracy.

Take Apple's control over the applications that can run on its devices (the article limits itself to programs that run on the iPhone or iPad, but even on their general computer line, the Mac, Apple is slowly clamping down—on my work-issued Mac laptop I had to dive deep into the settings so I could run programs I wrote myself on the laptop). They do it to enhance the user experience, going back as far as 1984 with the release of the first Macintosh. A major complaint at the time was the inability to modify the computer as there were no expansion slots. But on the flip side—it made the Macintosh easier to use and more stable.

Sure, you could add new hardware to a PC running MS-DOS, but it wasn't particularly easy to do (mainly due to a hardware limitation in the original IBM PC that other manufacturers followed to remain fully compatible wherein users had to learn about device addressing and IRQ settings) and there was always the chance that the device driver (a special program for talking to the hardware) for the device (and most likely not written by Microsoft) could have bugs that crashed the machine (at best—at worse it could silently corrupt memory). By fully controlling the hardware and restricting the upgrades, Apple could ensure a better user experience for The Rest Of Us™.

And now it's more than just hardware. Compter viruses (and yes, that is the proper plural for more than one computer virus) and worms are nothing new (The Shockwave Rider, a science fiction book first published in 1975, was partially about computer worms) but with the commercialization of the Internet in the mid 90s, the threat of malware has grown to such proportions (even images can be a vector for malicious computer code) that it makes sense to severely restrict what can run on a computer and restrict what the program can do (hence my having to tweak settings on my own laptop to allow me to run my own software). And Apple, by restricting the software that is allowed to run on their equipment, can curate the software, again, to make for a better user experience For The Rest Of Us™.

There's also a myth that large companies like Apple and Microsoft are trying to prevent The Rest Of Us™ from programming our own computers. During the rise of the home computer, the 70s and 80s, pretty much every computer sold came with some form of programming environment, even if the langauge was as simple as BASIC. But at the time, that was a selling point, primarily because there wasn't the large market for prewritten software that there is today. With the rise of shrinkware, there was less need to provide a programming environment with each computer.

And frankly, the number of people who buy computers and don't want to program outnumber the people who do want to program. By a lot (I attended high school in the mid 80s and took Pascal. I can probably count on one finger the number of people in that class who are still programming today. People, in general, don't want to program). There was pressure to lower the price of computers (there was a lot of competition in the 80s and 90s) and market research probably revealed that not many people cared about programming, and hey, if the customers don't care about BASIC, that's an easy thing to remove to lower the price. No, there's no vast conspiracy to keep people from programming, just a lack of interest.

I also run my own email server. I personally don't find it all that hard, but then again, I've been managing servers since the mid 90s and by now, I know the ins and outs of running an email server (harder these days than in the early 90s, long before spam and HTML laden emails deluged the Internet) but not many people want to bother with that, and I'm including people who could, in principle, run their own email servers. It's easier to leave that to a company that specializes in running email servers.

In fact, it's easier to rent computers in a data center than to own your own computers in a data center and leave the actual hardware up to companies that specialize in running them (and yes, I'm a person who rents a “computer” at a data center because it is cheaper and I don't have to bother with buying and managing the hardware, so even I am not completely immune to convenience). But I realize there's a risk with not having physical ownership of the hardware and for now, I can live with that risk.

But a vast conspiracy? Nah. I just don't see it.

Thursday, February 26, 2015

I feel the earth, move, under my feet

It's a bit disconcerting when the office is shaking. It would be one thing if I worked in The Los Angeles Office of The Corporation, but I'm don't—I work in The Ft. Lauderdale (as in Florida) Office of The Corporation (and besides, we don't have an office in Los Angeles).

I'm not alone in this—I asked around and yes, other people felt it too. It wasn't a violent “up and down and oh my God we're all going to die!” type of shaking, more of a “I can just make out this vibration in the floor; the building isn't going to collapse on us, is it?” type of vibration.

The only consolation (and it's not much) is that this isn't the first time this has happened.

And no one can explain why …

---

More unintentional conspiracies

Giving most people a “general purpose computer” these days is giving them enough rope to hang themselves. That’s why people that have never learned computers (or did and hate them) like iPads so much. It’s extremely difficult to mess anything up, and you don’t have to worry about antivirus and updating java and flash and all this other crap. Apps are sandboxed, privacy is taken seriously, background apps (spyware) can’t track you, etc.

As someone concerned with security, I’ll gladly tell people to switch from a virus-laden Windows laptop to an iPad or Chromebook.

As someone concerned with privacy, I’m conflicted in offering those suggestions because the security comes from proprietary app stores and review teams, trusting all your data to be stored by the GOOG, not having the ability to run your own code, etc.

Maybe it’s just as simple as: there is not one solution for everyone. Let the majority of people that have no interest in running their own code use iPads and Chromebooks. For developers and people that know enough to take precautions, keep using Macbooks and Thinkpads and whatever.

A story about Jessica | Lobsters

This is a comment on A Story About Jessica, and is presented here just as another data point about giving up control over our own computers.

Friday, February 27, 2015

Saving Private Data

We tell ourselves “once on the internet, always on the internet,” like maintaining content is a trivial thing. But it isn’t a trivial thing — at any time, the company that you rely on to keep your content for free could change their policies, or get bought out and change their policies, or decide they want to go public and change their policies, or simply go under and take your content with them.

The longevity of data requires more intent than this. My advice is to seriously consider migrating to a self-hosted site if you can. If you can’t make sure you export your data with some regularity.

Through some oh-so-very “Not Safe For Work” links (unless, of course, you work in the “adult industry,” in which case, the links are probably sfe) via Flutterby, Google Takes Back Adult Ban | /Slantist

If you consider your data important (and I think you should) and you are using a company to store (or manage) your data, then you must assume it can go away in an instant. No, really, the Internet is littered with dead companies that promised to keep your data “safe.” And you should keep you eye on the ones still alive, for there's no guarentee they'll be around tomorrow.

You might want to peruse The Archive Team while they're still around for more on this topic. Your data will thank you for it.

Saturday, February 28, 2015

A two inch cutting board on a table saw

Now that Bunny's brother has received his gift, I can talk about it. This past Monday on a whim, Bunny made this tiny cutting board:

The thing was, she made this with her table saw! You know, this huge monsterous thing with a 12″ or 14″ circular blade o'death spinning at a bazillion revolutions per second? Yeah, she made this teeny little cutting board with that.

But she made it to test out the capabilities of a new piece of equipment she bought—the Micro Jig GR-100 GRR-Ripper. It's a pretty amazing device used to push wood around and through the many spinning blades o'death that lurk in the wood shop (in our case, the garage). We've seen it used enough times (yes, we watch Steve Ramsey every week) that Bunny felt it might be worth getting.

And yes, it was worth getting. The demonstration videos that come with it are astounding (most impressive to me was cutting dados in a dowel). In fact, she was so impressed that she got a GRRR-Ripperfor her brother, and sent along the cutting board as an example of the type of work possible when using it.

• Current
• Next
• Previous
• First
• Last
• Top
• Home
• About
• Archive
• Search
• Glossary
• Copyright
• Help
• Accessibility

Obligatory Picture

Obligatory Contact Info

• Comments? sean@conman.org

Obligatory Feeds

• RSS Feed
• Atom Feed
• JSON Feed

Obligatory Links

• Flutterby!
• KIRK.is

Obligatory Miscellaneous

• About the Source Code
• Source Code

Obligatory AI Disclaimer

No AI was used in the making of this site, unless otherwise noted.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.