The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Saturday, October 03, 2009

Think

For Gregory, who will only use IBM ThinkPad laptops, comes the original IBM ThinkPad (link via Jason Kottke), which is still available for only a buck ninety-three.

But I want to know where to get this leather case for the IBM ThinkPad …

---

It's not a single-use brick, it's a lack of our imagination

Earlier last month, Jason Kottke posted a story about how Lego has become single use. It's the sort of golden-era thinking that I promised myself I wouldn't fall in to, but I ended up nodding along. Yeah, Lego's too corporate. Lego sold out!

Except that it hasn't.

Robin Sloan at Snarkmarket shook me out of my false nostalgia with the Tao of Lego. Despite opening by agreeing with Jason, Robin put together a post crammed to the gills with links to amazing repurposing of the supposedly single-use bricks. Want an example?

…

I bought a pile of the standard bricks and—as an experiment—this Star Wars kit to see how ridiculous the pieces were. On the box, it appears to be made of all-kinds of single-use bits. Building it told a different story. The feet of the walker turn out to be the same part as the bodies of the Droids. Some of the joints are re-purposed guns. There are dozens of little clever things so that as you follow the instructions, there is moment after moment of discovery. “Oh, I can do THAT with that part?”

Via Jason Kottke, There is no single-use Lego | Quiet Babylon

It's amazing what can be done using Lego, even with the non-Lego Lego Bionicle parts.

Click the link. Click all the links. You know you want to. And it's worth it.

Sunday, October 04, 2009

You know, had there been Pokémon cards when I was a kid, I might not have had to cheat in fourth grade math class …

He could go upstairs with his cousins to look over the cards and then pretend to be Pokémon characters for two solid hours—even though there was almost nothing else he could do, without distraction, for more than 20 minutes. Pokémon didn't seem so much an addiction as good-natured absorption—genuine, intrinsically oriented self-direction. We also realized the cards were teaching him category systems and math.

That following school year, in his first-grade class, Pokémon became social currency. About half his class was entranced by the cards. At times it seemed ridiculous, but then I'd hear my son plop down two cards and talk out more complicated math problems than anything he saw at school: “160HP minus 110HP plus 30 resistance points minus 20 weakness points equals 60 points left,” he'd say, then plop down two more cards to solve.

…

The second half of first grade, our son started reading the fine-print paragraphs on the cards. He got more reading time in through his love of Pokémon than he ever did at night, when we handed him books. He did read the books out loud to us, but it was a necessary chore. Pokémon was never a chore. And I noticed the paragraphs on the cards were syntactically far more complicated than anything he read in books. Soon, the same brain transformation that drove his math speed was reproduced with his reading speed.

Pokémon had taken over his brain. But in ways my wife never expected. Early in second grade, his math teacher told us he was as fast at math as the fifth graders. Not bad for a kid turned away by most of the local private schools prior to kindergarten.

Why Dumb Toys Make Kids Smarter - Page 1 - The Daily Beast

My mom used to tell me that at the end of my second day at kindergarten (or maybe it was 1^st grade) I was upset that I had not yet learned to read.

I'm not sure if that's what prompted her to let me pick and read comic books, or she just wanted for me to read anything I might enjoy, but up til the end of the 4^th grade, she would give me enough money to buy a few comic books every week (we moved to Florida just prior to my 5^th grade and I couldn't locate any nearby location that sold comic books, so that's pretty much when I stopped reading them). It must have helped some, because in 4^th grade I was placed into the 5^th grade reading class.

Tuesday, October 13, 2009

Going to the moon

“I hear you banging away at the keyboard,” said Bunny. “Will R like what you're working on?”

“Oh, I'm sure he'd like what I'm working on,” I said, “but whether he would pay me is another matter … ”

That was when I got The Look.

So, if I'm going to be banging away at the keyboard, might as well get a post out …

Over the past month, when I haven't been disinfecting websites and telling the people over at WC that no, their network is fine but we'll reinstall all the DNS servers anyway for Smirk and cramming in the hour here or there converting PHP from an undocumented hugely overwrought PHP framework to a documented and slightly overwrought PHP framework I've been playing around with Lua.

No, really!

I've been having a blast programming in a prototyped-based, dynamic, partially functional scripting language for the past month. I haven't had this much fun with a language since learning 68000 Assembly Language.

Yes, I know, I hate such languages, but over the past month, I've given it much thought as to why I like love Lua and hate the rest of the crowd.

Unlike Perl, which was designed by an insane linguist with a pediliction for punctuation who felt that a language of nothing but exceptions (much like spelling in the English language) would be fun, Lua is a simple language (heck, it's positively tiny—only 21 reserved words) that is very regular, with just a sprinkling of syntactic sugar to make it nice. Oh, and no sigils required on variables.

PHP is designed by rabid wombats—enough said.

Lua feels more like Python or JavaScript, only without the syntactic significance of whitespace and a consistant syntax for defining functions, both regular and anonymous, unlike Python, and has a much smaller code base than JavaScript, probably making it easier to embed.

And unlike Ruby, it's fast. I've found Lua to be faster than any of the other scripting languages I've mentioned so far.

It really comes down to a small, consistent scripting language that is easy to extend with C (or embed into a C/C++ application). And when I say small, I mean small. Twenty-one reserved words, twenty-six operators (like “(”, “)” or “‥”) and only 118 functions in the standard library, unlike the hundreds or thousands you'll find in the other languages.

Some people might not like the “batteries not included” feature of Lua, but since it's intended to be an embedded scripting language, that isn't much of a liability. You get a scripting language that allows you to declare variables, functions (both named and anonymous), closures, coroutines (or threads, if you want to think of them that way) and flow control. And I've been amazed at the extensions available for Lua.

So expect a deluge of Lua-related posts as I finally document what I've been doing for fun for the past month.

Sigh ‥ back to R's project and mucking with PHP …

Wednesday, October 14, 2009

Lua's long strings

I'm back from mucking with R's project for now …

One of the things I had to do on “Project: DoogieHowser” was figure out which tables needed updating before the questionnaire could start, in an attempt to get the thing as standalone as possible (since as written, it's almost, but not quite separate from the old undocumented hugely overwrought PHP framework) and one of the ways to test my findings was to populate the database and try it out.

And in order to test it over and over again, I wrote the code in PHP (this was just before I start learning Lua):

$now = time();
$id  = mysql_insert_id($conn);

$query = "INSERT INTO program ("
              . "order_number,"
              . "orderid,"
              . "clientid,"
              . "editable,"
              . "purchasedate,"
              . "reactive_date"
         . ") "
         . "VALUES ("
              . "'119195450271',"
              . "25278,"
              . $id . ","
              . "1,"
              . $now . ","
              . $now . ")";

$result = mysql_query($query,$conn);
$pid    = mysql_insert_id($conn);

… and what can I say, it's PHP. It's messy. It's nasty. It's got a pediliction for punctuation.

Now, one of the neatest features of Lua is the use of “long brackets”, which make the construction of huge strings trivial:

a_long_string = [[
`Twas brillig, and the slithy toves
  Did gyre and gimble in the wabe:
All mimsy were the borogoves,
  And the mome raths outgrabe.


"Beware the Jabberwock, my son!
  The jaws that bite, the claws that catch!
Beware the Jubjub bird, and shun
  The frumious Bandersnatch!"
]]

It's a brilliant solution to such a problem (and even neater, there's a method for embedding such long strings within a long string). It even extends to comments. In Lua, the line comment starts with a --:

-- frob the widget a few times, just to make sure
for i=1,10 do 
  frob(widget)
end

But a longer block comment starts with --[[ and ends with ]]:

--[[

We need to frob the widget multiple times, since the bitbucket sometimes get
stuck and needs to be forced a few times to clear up the issue.  Through
empirical testing we found that frobbing the widget 10 times will always
work, doesn't effect performance *all* that much, and much more importantly,
will allow me to get at least two hours sleep under the desk before da boss
comes in and starts screaming about deadlines ... 

]]

for i=1,10 do
  frob(widget)
end

So, had I used Lua, the above code would probably look something like:

now   = os.time()
id    = mysql.insert_id(conn)

query = string.format([[
	INSERT INTO program 
	(
		order_number,
		orderid,
		clientid,
		editable,
		purchasedate,
		reactive_date
	)
	VALUES 
	(
		'119195450271',
		25278,
		%s,
		1,
		%d,
		%d
	)]],id,now,now)

res = mysql.query(query,conn)
pid = mysql.insert_id(conn)

Not much different, but I didn't have to build up that huge string piecemeal, and it was certainly easier to type up.

And now that I think about it, I don't think it would be all that much code to fill in an SQL template with variables:

function sql_template(sql,text,vars)
  local function cmd(tag)
    local word = string.sub(tag,2,-2)
    
    if type(vars[word] == "string" then
      return sql.escape(vars[word])
    elseif type(vars[word] == "function" then
      return sql.escape(vars[word]())
    else
      return sql.escape(tostring(var[word]))
    end
  end

  local s = string.gsub(text,"{%w+}",cmd)
  return s
end

query = sql_template(mysql,[[
	INSERT INTO program 
	(
		order_number,
		orderid,
		clientid,
		editable,
		purchasedate,
		reactive_date
	)
	VALUES 
	(
		'119195450271',
		25278,
		{id}
		1,
		{now},
		{now}

	)]] , { id = mysql.insert_id(conn), now = os.time() });

Hmmm … I might have to see if I can do something like that in PHP; it would certainly help in “Project: Leaflet” …

---

A day in the life of the Googlenet

The following ticket comes in:

The warning message that my web site might be an attack site is showing up again. … The fact that I can get around the warning and get to the site doesn't alter the fact that it comes up sometimes instead of my home page. Business is bad enough without having customers scared away. It wasn't there yesterday. It is there again today.

This, and the phone conversation, lead me to believe that when he opens his browser, it immediately loads up the Google home page, and that he types his domain into the search box to get to his own site.

I say that because when I went to Google, and typed in his domain:

XXXX XXXXXX

This site may harm your computer.
Merchants, if you are interested in selling your wares in this market, call XXXXXXXXXXXX between 9-5 EST, Mon-Fri for information …
XXXXXXXXXXXXXXXXXXX - Similar

That, and I doubt he really understands how all this Intarweb stuff works, because that only happened on one of his three browsers.

Heck, for all I know, it could be the anti-virus software on the one computer gets some clues from Google (I'm not even sure how all this stuff works anymore, especially on the Microsoft Windows front).

Of course, his site uses quite a large amount of PHP … which meant I had to clean out the malicious PHP code that had been added …

I'm really of two minds about PHP. Personally, I hate the language (being designed by rabid wombats will do that for a language) and if I have a choice, I'd rather use C to write web applications than PHP (or maybe Lua). But I'll admit up front I'm a computer language snob who prefers early binding and static typing (comes from my Assembly Language background I'm sure), because PHP has allowed people who would otherwise be unable to create web applications not only the ability to create web applications, but find inexpensive hosting for said applications. Then again, it has allowed people who should otherwise be prevented from creating web applications the ability to create web applications more insecure than a balsa wood bank vault.

I think I'm digressing (and there's more I could say about insecure web applications, but that's for another post), but I'm not sure what point I want to make, but I think it has something to do with Google taking over the Internet. That, and the fact that because of the way they scale, they attempt to automate as much of the company as possible, and as a consequence, they have horrible customer service (and no, the irony of using Google to search for evidence of its own bad customer service isn't lost on me, nor the fact that I'm probably using the term “irony” incorrectly), so our customer will probably face an uphill battle to get “This site may harm your computer” off his Google search results.

---

A mass of stupid benchmarks. You have been warned.

A few weeks ago Wlofie and I were talking about computer languages and I mentioned the language I wrote in college (it was more than the simple DSL shown there), so I cleaned up the code so it would compile (most of it was written in 1993; last date of modification was March 5^th, 1995) and just because I was curious, I decided to run a stupid benchmark on it.

What's more stupid than just adding up the first 1,000,000 integers?

: main 
  0 0 
  begin 
    swap over + swap 1 + 
    dup 1000000 < while 
  repeat 
  drop "%d#n" string print ;
  
main 0 exit

(Oh, did I mention the language was based off Forth?)

Running that code on my modern 2.6GHz machine takes a full 2.1 seconds (and that's after I removed a huge bottle neck in the runtime engine). The equivalent C code:

#include <stdio.h>
#include <stdlib.h>

int main(int argc,char *argv[])
{
  int i;
  int total;
  
  for (i = 0 , total = 0 ; i < 1000000 ; i++)
    total += i;
  
  printf("%d\n",total);
  return EXIT_SUCCESS;
}

takes a mere 0.004 seconds to run.

Okay, that's expected—it's C. But what about Perl? 0.47 seconds.

Heh. A hundred times slower.

Hey! What about Python? It's not as fast as Perl, but it comes in at around 0.58 seconds.

I'm not even going to attempt a comparrison between my language and Ruby—it'd be too depressing to lose out to a known slow language.

But Wlofie did mention one other language he was playing around with—Lua. Okay, how fast is Lua? Same benchmark as above: 0.15 seconds.

Wow.

It blew Perl right out of the water.

This required a bit more investigation. And a better test than just summing up the first 1,000,000 integers. I settled upon the Jumble program. It involves I/O and has one non-trivial operation (sorting the letters in a word). And on my current system, the C version can check 483,523 words in 0.18 seconds (the timings I gave back in 2008 were for my older, 120MHz development machine).

The Lua code is straightforward:

#!/usr/local/bin/lua

WORDS = "/usr/share/dict/words"

-- ******************************************************

function sortword(word)
  local t = {}
  for i=1,string.len(word) do t[i] = string.byte(word,i) end
  table.sort(t)
  return string.char(unpack(t))
end

-- ********************************************************

if #arg < 1 then
  io.stderr:write("usage: %s word\n",arg[0])
  os.exit(1)
end

word = sortword(string.upper(arg[1]))

dict = io.open(WORDS,"r")
if dict == nil then
  io.stderr:write("Houston, we have a problem ... \n")
  os.exit(1)
end

for line in dict:lines() do
  dictsort = sortword(string.upper(line))
  if dictsort == word then
    print(line)
  end
end

dict:close()
os.exit(0)

sortword() has quite a bit of work to do—it has to go through the string, placing each letter into an array, sort the array, and then convert the array back into a string. Unlike C, which treats strings as an array of characters (and thus can be sorted without the data gymnastics), Lua (and pretty much all other modern scripting languages) treat strings at a single unit.

So it takes 7.9 seconds for Lua to run through the same 483,523 words.

Ouch. And I think it's pretty obvious where the time is being spent—in sortword(), pulling apart strings, sorting arrays, making new strings. But Lua can be easily extended using C. I rewrite sortword() in C:

#include <stdlib.h>
#include <string.h>

#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>

/*******************************************/

static int cmp(const void *l,const void *r)
{
  int cl;
  int cr;
  
  cl = *(const char *)l;
  cr = *(const char *)r;
  
  return cl - cr;
}

/************************************/

static int sortword(lua_State *L)
{
  const char *word;
  size_t      len;
  
  word = luaL_checkstring(L,1);
  len = strlen(word);
  
  char buffer[len];
  memcpy(buffer,word,len);
  buffer[len] = '\0';
  qsort(buffer,len,1,cmp);
  lua_pushlstring(L,buffer,len);
  return 1;
}

/***********************************/

int luaopen_sortword(lua_State *L)
{
  lua_register(L,"sortword",sortword);
  return 0;
}

And now it only takes 2.2 seconds. Not a bad return upon investment.

Now we turn to Perl:

#!/usr/bin/perl
use strict;

#***************************

sub sortword
{
  my $str = shift;
  my @t   = unpack("(C)*",$str);
  
  @t = sort(@t);
  return pack("(C)*",@t);
}

#*******************************

if (scalar(@ARGV) == 0)
{
  printf(STDERR "usage: %s word\n",$0);
  exit(1);
}

my $word = sortword(uc($ARGV[0]));

open(DICT,"/usr/share/dict/words") 
  or die("Houston, we have a problem ... \n");

while(my $line = <DICT>)
{
  chop $line;

  my $dictsort = sortword(uc($line));
  
  if ($dictsort eq $word)
  {
    print $line . "\n";
  }
}

close(DICT);
exit(0);

Again, we need to pull apart the string into an array, sort that, and recreate the string. And when we run this lovely bit of Perl code it only takes 10.5 seconds.

Wow.

And here I thought Perl was the text crunching Master of the Universe.

Guess not.

Well, the Perl version of sortword() could be tightened up a bit …

sub sortword
{
  return pack("(C)*",sort(unpack("(C)*",shift)));
}

That takes us down to 8.6 seconds—nearly two full seconds to shuffle data between variables. Perl isn't looking all that good. Well, there is the function overhead, and given that we got sortword() down to one line, maybe if we inline it we'll get an improvement.

And we do—it's not 7.2 seconds. We finally beat unoptimized Lua code, but now with a more cryptic Perl version.

Way to go, Perl!

But given that Lua was easy to extend with C, can we do the same with Perl? A bit of searching lead me to the Inline::C Perl Module. Ten minutes to download and install (much better than I expected) and if anything, it was easier to embed C in Perl than in Lua:

#!/usr/bin/perl
use Inline C;
use strict;

# same code as above, minus the sortword() subroutine

exit(0);

__END__
__C__

static int cmp(const void *l,const void *r)
{
  int cl;
  int cr;
  
  cl = *(const char *)l;
  cr = *(const char *)r;
  
  return cl - cr;
}

char *sortword(char *word)
{
  size_t len;

  len = strlen(word);
  qsort(word,len,1,cmp);
  return word;
}

And we're down to 2.1 seconds. Just a bit quicker than the Lua/C version.

Well, that's after the 4.1 second initial run where it dumped a bunch of directories and files (lovely that).

Lua appears to be a bit faster than Perl, or at the very least, as fast as Perl.

And everything (even, quite possibly, Ruby) is faster than my own homebrewed language … sigh.

Friday, October 16, 2009

”Red Alert!” “Where? I don't see any lerts around here!”

Yesterday felt like Friday, mainly because The Weekly Meeting™ was canceled at the last minute, thus I didn't have to leave The Home Office.

But it was still a weekday today when I got a call rather early from Smirk. “Sean!” he yelled through the phone into my ear, “not only do we have a customer down, but we have network traces that show it's our equipment that's down!” Behind him I could hear the Klaxons™ in the background.

I mumbled something about calling him back, wandered over to The Home Office, and started a Level Five Diagnostic Program poking into the routers.

Oct 16 11:06:59.490 EDT: %RSP-3-ERROR: MD error 0080000080000000 -Traceback= 60385460 60385B24 60385C48 60386614 603513F4
Oct 16 11:06:59.494 EDT: %RSP-3-ERROR:  SRAM parity error (bytes 0:7) 80 -Traceback= 60385538 60385B24 60385C48 60386614 603513F4
Oct 16 11:06:59.494 EDT: %VIP2 R5K-3-MSG: slot1 VIP-3-SVIP_CYBUSERROR_INTERRUPT: A Cybus Error occured.
Oct 16 11:06:59.498 EDT: %VIP2 R5K-1-MSG: slot1 CYASIC Error Interrupt register 0xC
Oct 16 11:06:59.502 EDT: %VIP2 R5K-1-MSG: slot1   Parity Error internal to CYA
Oct 16 11:06:59.506 EDT: %VIP2 R5K-1-MSG: slot1   Parity Error in data from CyBus
Oct 16 11:06:59.514 EDT: %VIP2 R5K-1-MSG: slot1 CYASIC Other Interrupt register 0x100
Oct 16 11:06:59.518 EDT: %VIP2 R5K-1-MSG: slot1   QE HIGH Priority Interrupt
Oct 16 11:06:59.522 EDT: %VIP2 R5K-1-MSG: slot1   QE RX HIGH Priority Interrupt
Oct 16 11:06:59.526 EDT: %VIP2 R5K-1-MSG: slot1 CYBUS Error Cmd/Addr 0x8001A00
Oct 16 11:06:59.530 EDT: %VIP2 R5K-1-MSG: slot1 MPUIntfc/PacketBus Error register 0x0
Oct 16 11:06:59.534 EDT: %VIP2 R5K-3-MSG: slot1 VIP-3-SVIP_PMAERROR_INTERRUPT: A PMA Error occured.
Oct 16 11:06:59.538 EDT: %VIP2 R5K-1-MSG: slot1 PA Bay 0 Upstream PCI-PCI Bridge, Handle=0
Oct 16 11:06:59.542 EDT: %VIP2 R5K-1-MSG: slot1 DEC21050 bridge chip, config=0x0
Oct 16 11:06:59.546 EDT: %VIP2 R5K-1-MSG: slot1 (0x00):dev, vendor id    = 0x00011011
Oct 16 11:06:59.550 EDT: %VIP2 R5K-1-MSG: slot1 (0x04):status, command   = 0x02800147
Oct 16 11:06:59.554 EDT: %VIP2 R5K-1-MSG: slot1 (0x08):class code, revid  = 0x06040002
Oct 16 11:06:59.562 EDT: %VIP2 R5K-1-MSG: slot1 (0x0C):hdr, lat timer, cls = 0x00010000
Oct 16 11:06:59.566 EDT: %VIP2 R5K-1-MSG: slot1 (0x18):sec lat,cls & bus no = 0x08010100
Oct 16 11:06:59.570 EDT: %VIP2 R5K-1-MSG: slot1 (0x1C):sec status, io base = 0x22807020
Oct 16 11:06:59.574 EDT: %VIP2 R5K-1-MSG: slot1     Received Master Abort on secondary bus
Oct 16 11:06:59.578 EDT: %VIP2 R5K-1-MSG: slot1 (0x20):mem base & limit   = 0x01F00000
Oct 16 11:06:59.582 EDT: %VIP2 R5K-1-MSG: slot1 (0x24):prefetch membase/lim = 0x0000FE00
Oct 16 11:06:59.586 EDT: %VIP2 R5K-1-MSG: slot1 (0x3C):bridge ctrl     = 0x00030000
Oct 16 11:06:59.590 EDT: %VIP2 R5K-1-MSG: slot1 (0x40):arb/serr, chip ctrl = 0x00100000
Oct 16 11:06:59.594 EDT: %VIP2 R5K-1-MSG: slot1 (0x44):pri/sec trgt wait t. = 0x00000000
Oct 16 11:06:59.598 EDT: %VIP2 R5K-1-MSG: slot1 (0x48):sec write attmp ctr = 0x00FFFFFF
Oct 16 11:06:59.606 EDT: %VIP2 R5K-1-MSG: slot1 (0x4C):pri write attmp ctr = 0x00FFFFFF
Oct 16 11:06:59.610 EDT: %VIP2 R5K-1-MSG: slot1 PA Bay 1 Upstream PCI-PCI Bridge, Handle=1
Oct 16 11:06:59.614 EDT: %VIP2 R5K-1-MSG: slot1 DEC21050 bridge chip, config=0x0
Oct 16 11:06:59.618 EDT: %VIP2 R5K-1-MSG: slot1 (0x00):dev, vendor id    = 0x00011011
Oct 16 11:06:59.622 EDT: %VIP2 R5K-1-MSG: slot1 (0x04):status, command   = 0x02800147
Oct 16 11:06:59.626 EDT: %VIP2 R5K-1-MSG: slot1 (0x08):class code, revid  = 0x06040002
Oct 16 11:06:59.630 EDT: %VIP2 R5K-1-MSG: slot1 (0x0C):hdr, lat timer, cls = 0x00010000
Oct 16 11:06:59.634 EDT: %VIP2 R5K-1-MSG: slot1 (0x18):sec lat,cls & bus no = 0x08020200
Oct 16 11:06:59.638 EDT: %VIP2 R5K-1-MSG: slot1 (0x1C):sec status, io base = 0x2280F0A0
Oct 16 11:06:59.642 EDT: %VIP2 R5K-1-MSG: slot1     Received Master Abort on secondary bus
Oct 16 11:06:59.650 EDT: %VIP2 R5K-1-MSG: slot1 (0x20):mem base & limit   = 0x03F00200
Oct 16 11:06:59.654 EDT: %VIP2 R5K-1-MSG: slot1 (0x24):prefetch membase/lim = 0x0000FE00
Oct 16 11:06:59.658 EDT: %VIP2 R5K-1-MSG: slot1 (0x3C):bridge ctrl     = 0x00030000
Oct 16 11:06:59.662 EDT: %VIP2 R5K-1-MSG: slot1 (0x40):arb/serr, chip ctrl = 0x00100000
Oct 16 11:06:59.666 EDT: %VIP2 R5K-1-MSG: slot1 (0x44):pri/sec trgt wait t. = 0x00000000
Oct 16 11:06:59.670 EDT: %VIP2 R5K-1-MSG: slot1 (0x48):sec write attmp ctr = 0x00FFFFFF
Oct 16 11:06:59.674 EDT: %VIP2 R5K-1-MSG: slot1 (0x4C):pri write attmp ctr = 0x00FFFFFF
Oct 16 11:06:59.678 EDT: %VIP2 R5K-3-MSG: slot1 VIP-3-SVIP_RELOAD: SVIP Reload is called.
Oct 16 11:06:59.690 EDT: %VIP2 R5K-3-MSG: slot1 VIP-3-SYSTEM_EXCEPTION: VIP System Exception occurred sig=22, code=0x0, context=0x60A8D368
Oct 16 11:07:01.714 EDT: %DBUS-3-CXBUSERR: Slot 1, CBus Error
Oct 16 11:07:01.714 EDT: %DBUS-3-DBUSINTERRSWSET: Slot 1, Internal Error due to VIP crash
Oct 16 11:07:01.718 EDT: %RSP-3-ERROR: End of MEMD error interrupt processing -Traceback= 60385BF0 60385C48 60386614 603513F4
Oct 16 11:07:01.842 EDT: %DBUS-3-CXBUSERR: Slot 1, CBus Error
Oct 16 11:07:01.842 EDT: %DBUS-3-DBUSINTERRSWSET: Slot 1, Internal Error due to VIP crash
Oct 16 11:07:05.599 EDT: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0x5800FF20, slot 0, cmd code 2 -Traceback= 603E3AF8 603E3FA4 603DC230 603D9F70 603A1E8C 6034462C 602FA6F0 603241C8 603241B4
Oct 16 11:07:07.623 EDT: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to down
Oct 16 11:07:08.687 EDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
Oct 16 11:07:16.527 EDT: %RSP-3-RESTART: cbus complex

I've never seen that happen to a Cisco router before.

I called Smirk back. “Smirk, you better get G [our Cisco consultant —Editor] on the phone to diagnose this issue. I'm out of my league.”

And thus began a few hours of scrambling to get a replacement router for the customer, and by the time I got onsite with a temporary replacement, I was told it was too late to do the change (the current router was still limping along—these crashes were happening about every half hour) and that they were planning on taking down the network at 11:00 tomorrow, so I could do the replacement then.

Wonderful!

Worse, is that this isn't the first time this router has had problems. A few weeks ago we appeared to have a similar issue and ended up replacing one of the interfaces (the errors weren't nearly as scary then). I remarked at the time that I had never seen a Cisco router go bad (I've been working with Smirk at The Company for five years now and this is a first—and even when I worked at a webhosting company in the late 90s, never saw a bad Cisco router, nor did I come across one when working at two ISPs (one in the mid 90s, and one around the turn of the century). Smirk also informed the customer, multiple times since then, that they need a redundant router, but the request never made it past a certain level of management.

Sigh.

Saturday, October 17, 2009

Them ol' router blues …

Getting up early makes me surly and not pleasant to be around. Stress also makes me surly and not pleasant to be around.

So of course, I'm getting up early to walk into a stressful situation.

I thought the plan was to simply drop in the Cisco 3550 (which I had configured for use yesterday) until we could get a full replacement router, but alas, that was not to be. Smirk called just before I left and told me to grab the Cisco 7206 and salvage a few cards from the 7500 series (I forget the exact model) over at the customer site.

So much for “get in, get out, nobody gets hurt” routine.

So I have to swing by The Data Center in Boca Raton first and grab the 40# beast, thus making me a bit late over at the Customer Site.

It didn't matter much because this is what greeted me when I arrived:

Thankfully, Smirk is used to my surliness at the crack of dawn [It was 11:03 am. –Editor][For me, 11:00 am is the crack of dawn. —Sean] and ignored my yelling on the phone. I was told to sit tight and I'd be let in momentarily.

I'll spare you the next three hours, but basically, they weren't happy with the Cisco 3550 (“What? You expect us to use that? Cisco ‘end-of-lifed’ that! That's not acceptible”) nor were they (or I) happy with the speed the Cisco 7206 was getting (directly connected, I was getting 90Mbps upload and only 10Mbps download speed—they were getting worse across their network. The Cisco 3550 (despite being “end-of-lifed” by Cisco) was faster. Smirk told me to take a break at 2:00, and meet him back at the customer site at 5:00 pm.

Some food and relaxation smoothed out the surliness on my part. I met Smirk back at the customer site at 5:00 pm and over the next three and a half hours:

1. we wasted probably half an hour or more because a cable went bad;
2. even though the bottom interface (the main CPU card) on the 7206 was labeled “FastEthernet” (Ciscospeak for “100Mbps port) it wasn't. Sure, it could communicate with a device at 100Mbps but it couldn't sustain that speed (which explains the huge discrepency with the earlier speed tests);
3. cards without a handle are next to impossible to pull out of a Cisco (oh, that was an ugly scene);
4. you need to load the cards into a Cisco 7206 in slot order;
5. the customer's network disallowed pinging (nice one—remove one of the best network diagnostic tools why don't you?);
6. spent an additional hour with the customer crew when we proved the connection was getting 90Mbps up and down so they could track down the speed bottleneck on their network.

Hopefully now they'll listen and get the redundant routers Smirk has been telling them they should get.

Monday, October 19, 2009

Chlorinate your pool without chlorine

Mark was eager to show me his latest toy that was just installed at his house—a chlorinator. I'm sure his having a hand in the development lead to part of his enthusiam for the Saltron™.

I'll admit, never having seen one of these units before, it's quite cool how it works. It's hooked inline with the pool pump, and converts ordinary salt (sodium chloride—the stuff you put on food) that is dumped into the water into chlorine.

The process uses an electrochemical process to extract the chlorine from the salt. The unit where the conversion takes place does get dirty over time, but it's designed to be removed and cleaned using an acid bath to remove the scales that build up over time. The computer controller that Mark helped develop reverses the polarity of the device every four hours since the scales develop on the negatively charged end—this is to keep the deposit buildup fairly consistent over the entire device making it last longer between cleanings (or replacements).

The unit is still being tested and Mark has one of the 50 prototype units, and apparently, the company that makes them is still looking for more testers. If anyone reading this is interested, I can forward your request to Mark.

Wednesday, October 21, 2009

One Moon Daemon, coming up

My latest adventure into Lua has been to embed the language into a C application (an outgrowth of extending in C). I had this “proof-of-concept” network daemon I wrote that was simple enough to gut completely and replace the entire server logic with Lua, leaving the C code to handle the sockets.

But that wasn't hard enough. No. I wanted each connection handled by a Lua thread, and the ability to write “normal looking code” such as:

-- A simple echo server in Lua

function main(connection)
  while true do
    connection:write(connection:read("*a"))
  end
end

I will say this, unlike Theo Schlossnagle, who doesn't care for Lua the language but likes embedding it, I came to the opposite conclusion: I like Lua the language but embedding it is … interesting. Perhaps because I dove headfirst into Lua coroutines to get this working.

The first oddness with embedding Lua is that you push parameters onto the Lua stack, but not all lua functions pop all parameters. Namely, anything using a table won't actually pop the table parameter off the stack, and the stack notation used in the manual is not the notation I'm used to (having come to stack notations in Forth). So I'm sure that a majority of the issues I had were due to improper stack manipulations on my part.

And about that stack. You can reference the stack in two ways, from the bottom using positive indices, or from the top using negative indices, with 0 as an invalid stack index.

Lua execution stack of five elements

Bottom of Stack				Top of Stack
1 -5	2 -4	3 -3	4 -2	5 -1

As a longtime C (and assembly) programmer, it just looks weird to use 1- based indices in C.

Another horrible issue—via the C API you can push integers onto the Lua stack, you can push strings onto the Lua stack, you can push functions (either written in Lua or C) onto the stack, heck, you can push everything but userdata structures onto the Lua stack. Userdata structures are allocated by Lua (via lua_newuserd ata()), can contain anything you want, but there's no way to actually push such a structure onto the Lua stack, so you always have to keep it around on the Lua stack, or in a Lua variable or table, which is a bit of a pain. I suppose it has to be that way in order for Lua to keep track of it for garbage collection purposes, but still, a pain.

And then there's the coroutines …

Easy enough to create a coroutine:

lua_State *new;
Foo       *userdata;
int        rc;

/*----------------------------
; create a new thread 
; main(userdata)
;----------------------------*/

new      = lua_newthread(g_L);
lua_getglobal(new,"main");
userdata = lua_newuserdata(new,sizeof(Foo));
	/* fill in our userdata */
rc = lua_resume(new,1);

But once the thread finishes, then what?

lua_close

void lua_close (lua_State *L);

Destroys all objects in the given Lua state (calling the corresponding garbage-collection metamethods, if any) and frees all dynamic memory used by this state. On several platforms, you may not need to call this function, because all resources are naturally released when the host program ends. On the other hand, long-running programs, such as a daemon or a web server, might need to release states as soon as they are not needed, to avoid growing too large.

Lua 5.1 Reference Manual

Okay, so I'm concerned about garbage collection—I don't want tons of garbage piling up and then … the … … big … … … pause … … … … while … … … … … garbage … … … … … … collection takes place. And seeing how lua_close() mentions daemons, I figure I can call lua_close when a thread terminates, since I'm writing a daemon. But when I do, every open connection suddenly closes and any new connection causes the daemon to crash.

Huh?

lua_newthread

lua_State *lua_newthread (lua_State *L);

Creates a new thread, pushes it on the stack, and returns a pointer to a lua_State that represents this new thread. The new state returned by this function shares with the original state all global objects (such as tables), but has an independent execution stack.

There is no explicit function to close or to destroy a thread. Threads are subject to garbage collection, like any Lua object.

Lua 5.1 Reference Manual

Reading between the lines, you don't really get a new state, you get a copy of the existing state, but a new execution stack. Calling lua_close() on this “new” state is just like calling lua_close() on the already existing state.

Nope. What I wanted to do after each thread was to call lua_gc(), only now current connections are getting garbage-collected along with the finished ones, which meant I wasn't using luaL_ref() and luaL_unref() properly, probably because I wasn't managing the Lua stack properly …

Sigh.

It also took a bit of careful coding to get the Lua code to properly block (lua_yield()) and resume (lua_resume()) when you have input and output asynchronously appearing. It was also a bit tricky to have one Lua thread write to another Lua thread's socket without blowing things up. But eventually, not only did I get it such that the echo service above works (as written above) but the following simple chat script as well:

if members == nil then
  members = {}
end

local function login(socket)
  socket:write("Handle you go by: ")
  return socket:read()
end

local function wallaction(socket,who,everybody,me)
  for connection in pairs(members) do
    if connection ~= socket then
      connection:write(string.format("%s %s\n",who,everybody))
    else
      if me ~= nil then connection:write(string.format("%s\n",me)) end
    end
  end
end

local function wall(socket,who,everybody,me)
  return wallaction(socket,who .. ":",everybody,me)
end

function main(socket)
  local name      = login(socket)
  members[socket] = name
  
  wallaction(socket,name,"is in da room!","You are in the room.")
  io.stdout:write(string.format("%s has joined the party!\n",name))
  
  while true do
    wall(socket,name,socket:read())
  end
end

function fini(socket)
  io.stdout:write(string.format("%s has left the party!\n",members[socket]))
  wallaction(socket,members[socket],"has left the building!")
  members[socket] = nil
end

Each Lua thread starts with main(). fini() is called when a connection is dropped. I also made the daemon such that sending it a SIGUSR1 will cause it to re-compile the script, so that changes to the service can be made without having to restart the program as a whole (which explains the odd way I define members—if we reload the script, I don't want to lose the current members).

As written, the code isn't multi-threaded—at the C level, it's just one process round-robinning a bunch of Lua threads, so I can get away with the chat process above without worry as only one Lua thread is executing at any one time.

But it's now quite easy to write network daemons, so easy that I'll leave you with one more—an RFC compliant Quote of the Day server:

QUOTESFILE = "/home/spc/quotes/quotes.txt"
quotes     = {}

do
  local eoln = "\r\n"
  local f    = io.open(QUOTESFILE,"r")
  local s    = ""

  for line in f:lines() do
    if line == "" then
      -- each quote is separated by a blank link
      if #s < 512 then
        table.insert(quotes,s)
      end
      s = ""
    else
      s = s .. line .. eoln
    end
  end

  f:close()
end

math.randomseek(os.time())

function main(socket)
  socket:write(quotes[math.random(#quotes)])
end

Thursday, October 22, 2009

Unicorns and Rainbows

I don't use Drupal. I don't care for PHP. Yet I still ended up attending a Drupal User Group because both Smirk and R use Drupal (and I guess that means I just outted the documented overwrought PHP framework he's switching to) and that means I too, have to quaff the non-cabonated sugar water.

Yum.

The first presentation was on what looked like a framework built ontop of Drupal, which is a framework unto itself, and I was thinking that the whole point of Drupal was to provide a framework to remove the drudgery of website construction yet it's now too low level and cumbersome to work with so let's wrap the whole thing up in yet another obtuse abstraction layer to further remove us from any tangible implementation details.

Yeah.

[In talking with Smirk after the meeting, I learned that what was being described with breathless enthusiasm was really just a Drupal distribution with a bunch of pre-installed third party modules installed for constructing a particular type of website. I'm glad Smirk cleared up my misconception of that presentation.]

The second presentation was on drush, which is a commnd line interface to administer a Drupal site because the Drupal control panel is apprently just too limited. Everybody (including Smirk) was oohing and ahhing over this incredible new approach to adminstration and I'm like … this is new? Am I missing something here? (short answer for now: yes, but I'll get to that in a bit)

The final presentation was on the upcoming Drupal 7 release, which includes a bunch more modules in the core distribution, “stories” are now called “articles,” and a few minor API changes (but don't worry—there's a project to automatically convert 6x modules to 7).

If I seem underwhelmed, it's … well, because I am. I was not in the target audience for the meeting and in fact, I seemed to be the only programmer in attendance [let me clarify that a bit: while Smirk can program—he has a Computer Science degree and is getting his Masters; I think I was the only person there who actually makes a living by programming. Everbody else doesn't and even those that could program stated they will spend extra time to avoid programming if at all possible—why spend five hours writing a custom Drupal module that works exactly as you need when you can spend ten hours trying to find a module that sort of, kind of, does almost what you want so you can work around it's limitations, because, as we all know, everything that you could possibly do on the computer has already been done before, so why reinvent those square granite wheels?] [Cynical much? —Editor] [What? Me, cynical? Naaaah … —Sean]

Between the presentations and hanging out listening to the conversation afterwards, I felt very out of place there. One person there was going on and on about Titanium, a development environment that allows one to make “desktop applications” using only HTML5, CSS3 and JavaScript that can be installed everywhere (that is, under Windows, Mac OS-X and Linux). That's not programming as I know it (and why was he bringing it up at a Drupal User's Group? I don't know).

Another person was complaining about slow websites, and when I mentioned the easy solution to that (just serve up static pages) he just rolled his eyes and proceeded to continue on with the conversation (he came across as nice, but as he said, he wasn't one of those “server guys.” Sigh).

I had a hard time explaining my reaction to Smirk after the meeting. I mean, I can understand why Smirk is using Drupal to design some websites, as it allows the customer the ability to log in and customize their site without having to bother us, but as I attempted to explain to Smirk, this ever increasing layering of abstractions is all unicorns and rainbows until something breaks.

And now what?

Did the router go down? Is DNS borked? Oh wait, Apache could have run out of file descriptors and thus can't load the site any more. Could be a disk quota issue, then again, it could be that a file that the web application locked was on an NFS mounted drive that actually was symlinked to a file shared via SMB from a Windows machine in accounting and Bob just changed the contents of the file out from under the web application without knowing it.

But I digress.

I'm not saying I had a horrible time, I didn't. It wasn't that bad. I just wasn't the intended audience. And the anti-programming extreme user-centric vibe in the room didn't help either.

I'm also thinking I need to do a post on unicorns and rainbows needlessly complex abstractions …

Friday, October 23, 2009

WikiMUD Part Deux: Five Years Later …

From

Giles Cooper <XXXXXXXXXXXXXXXXXXXXXX>

To

sean@conman.org

Subject

WikiMUD

Date

Fri, 23 Oct 2009 16:03:24 -0400

… (almost exactly five years later) …
Go on, I'm entrigued

– 
Giles Cooper
XXXXXXXXXXXXXXXXXXXXXX
http://planes09.wikia.com/

Giles is refering to two entries I wrote five years ago. It was more of a passing idea than anything I really wanted to work on and over the years I felt that the most likely language to implement the WikiMUD would be JavaScript (as it's pretty much Lisp in imperative form … kind of … if you squint real hard … and wave your hands around a lot), but over the past month or so, I'm thinking Lua would be a good candidate as well.

Especially given that I can now write network services in Lua.

Okay, doesn't mean I'll actually write a WikiMUD … just that now I'm in a better position to start, if I decide to.

Or someone can …

Or something …

---

I'm sensing a trend …

In June, Al Byrd's three-bedroom home, built by his father on the western outskirts of Atlanta, was mistakenly torn down by a demolition company. “I said, ‘Don't you have an address?’ ” a distraught Byrd later recounted. “He said, ‘Yes, my GPS coordinates led me right to this address here.’ ” The incident joined a long list of satellite-guided blunders, including one last year in which a driver in Bedford Hills, New York, obeyed instructions from his GPS to turn right onto a set of train tracks, where he got stuck and had to abandon his car to a collision with a commuter train. Incredibly, the same thing happened to someone else at exactly the same intersection nine months later. In Europe, narrow village roads and country lanes have turned into deadly traps for truckers blindly following GPS instructions, and an insurance company survey found that 300,000 British drivers have either crashed or nearly crashed because of the systems.

…

To many, the beauty of the devices is precisely that we no longer have any need to painstakingly assemble those cognitive maps. But Cornell University human-computer interaction researcher Gilly Leshed argues that knowledge of an area means more than just finding your way around. Navigation underlies the transformation of an abstract “space” to a “place” that has meaning and value to an individual. For the GPS users Leshed and her colleagues observed in an ethnographic study, the virtual world on the screens of their devices seemed to blur and sometimes take over from the real world that whizzed by outside. “Instead of experiencing physical locations, you end up with a more abstract representation of the world,” she says.

On a snowmobile trip of over 500 kilometres across the Arctic, this blurring of the real and the virtual became obvious to Carleton University anthropologist Claudio Aporta. Returning from Repulse Bay to Igloolik, a village west of Baffin Island where he was conducting fieldwork, he and an Inuit hunter became engulfed in fog. The hunter had been leading the way along traditional routes, guided by the winds, water currents, animal behaviour, and features such as the uqalurait, snowdrifts shaped by prevailing winds from the west by northwest. Like London taxi drivers, Inuit hunters spend years acquiring the knowledge needed to find their way in their environment, part of a culture in which “the idea of being lost or unable to find one's way is without basis in experience, language, or understanding — that is, until recently,” as Aporta and Eric Higgs wrote in a 2005 paper on “satellite culture” and the rise of GPS use in Igloolik.

Heavy fog is the one condition that stymies even the most expert Inuit navigators. The traditional response is to wait until the fog lifts, but, knowing that Aporta had mapped the outbound journey on his GPS, his guide asked him to lead the way on his snowmobile. “It was an incredible experience, because I could see absolutely nothing,” he recalls. “I didn't know if there was a cliff ahead; I was just following the GPS track for five kilometres, blind, really.” This was the extreme version of the city driver blankly turning left and right at the command of his GPS, and it required a leap of faith. “Believe me,” he says, chuckling, “I was sweating like crazy.”

The demonstrable benefits of GPS have, however, removed much of the incentive for the younger generation in Igloolik to undertake the arduous process of learning traditional navigation techniques. Elders worry about this loss of knowledge, for reasons that go beyond the cultural—a straight line across an empty icefield plotted by GPS doesn't warn about the thin ice traditional trails would have skirted. Dead batteries and frozen screens, both common occurrences in the harsh Arctic conditions, would also be disastrous for anyone guided solely by technology.

Via Hacker News, The Walrus Magazine » Global Impositioning Systems

I think there's a connection between overreliance on the GPS and my unease with the Drupal User's Group yesterday, but it's still a bit tenuous … but there is a connection …

And one story somewhat related to the article: when I visited my friends in Boston, I had no sense of the city (other than being a twisty maze of one way streets, all alike) because we always took the T, which was for the most part, below ground. We'd descend into an underground station, enter a train, wait a bit, exit the train and ascend into a new part of the city—a linear stretch of Bostonian islands as it were.

I found it rather disconcerting, but I never did get lost.

• Current
• Next
• Previous
• First
• Last
• Top
• Home
• About
• Archive
• Search
• Glossary
• Copyright
• Help
• Accessibility

Obligatory Picture

Obligatory Contact Info

• Comments? sean@conman.org

Obligatory Feeds

• RSS Feed
• Atom Feed
• JSON Feed

Obligatory Links

• Flutterby!
• KIRK.is

Obligatory Miscellaneous

• About the Source Code
• Source Code

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.