The Boston Diaries

Wednesday, April 01, 2015

I solemnly swear that on this April Fools' Day I will not post a fake engagement photo on MyFaceGoogleBookPlusSpace like some kind of asshole. Nor will I perpetuate a celebrity death hoax. Because that makes people sad. In fact, I will not play any practical jokes. And if I see someone planning a prank, I will say to them: “Hey, you're being a dick right now. Stop being a dick. Stop being a dick. Seriously, stop being a dick.”

Seriously, take the pledge. The rest of the world will thank you for it.

Serializing Lua data, part III

Continuing the series on Lua serialization, we resume with the Lua userdata type. At this point, all I'm doing is conjecture, since I haven't actually bothered with serializing userdata and this post is about why I didn't bother.

The Lua manual has this to say about userdata:

The type userdata is provided to allow arbitrary C data to be stored in Lua variables. A userdata value represents a block of raw memory. There are two kinds of userdata: full userdata, which is an object with a block of memory managed by Lua, and light userdata, which is simply a C pointer value. Userdata has no predefined operations in Lua, except assignment and identity test. By using metatables, the programmer can define operations for full userdata values (see §2.4). Userdata values cannot be created or modified in Lua, only through the C API. This guarantees the integrity of data owned by the host program.

Already we're in trouble with respect to serialization. The light userdata is just a pointer, no more, no less. There is no indication of just how much memory it points to, much less what it points to or what the memory even represents. That's not to say that a light userdata has no meaning, but such meaning is only applicable to any code written in C (or C++) and not to Lua.

The serialization code gets a light userdata and … meh? There is nothing we can safely serialize about a light userdata. One answer might be to serialize the pointer value, but a pointer value only has meaning within an instance of a program. Run the same program again, and the value of the light userdata might not be the same (nor should you assume it'll be the same). Give that pointer value to another program, and who knows what it points to in that other program.

We could try reading the memory the light userdata points to, but as I said, there is no indication of how much memory we can read, if any. In general, serialization of light userdata is out of the question (I'm not saying it can't be done, I'm just saying I'm not writing a general purpose serializer that will attempt it).

A full userdata, while it still has no meaning for Lua, does have a length associated with it, so it is conceivable we could serialize the actual memory associated with a full userdata but again, we run into trouble. For instance, let's try to serialize the only standard Lua userdata—the result of opening up a file.

The userdata associated with a file in Lua is the following structure:

typedef struct luaL_Stream {
  FILE *f;
  lua_CFunction closef;
} luaL_Stream;

The first field is a pointer to a FILE, an opaque type (even to C programmers) that represents an “open file,” and sending the actual FILE value across is meaningless; one, because the contents differ from system (say, Microsoft Windows) to system (Linux), and two, it supposes that the “file” being opened exists when this is deserialized and used.

Now, given that CBOR allows additional semantics to be attatched to data, we could include the contents of the “open file” as part of the serialization and semantically tag the data as a “file, please create and open this.” We could, but there are a few issues even with this—what if the “open file” isn't a file at all? What if it's the output from a command? Or a network connection? How you do serialize a connected TCP connection? But say that we do have, in fact, a file, one that happens to be two gigabytes in size? Would you really want to serialize a two gigabyte file?

So solve those issues, there's still the issue of how the file was opened. Was it opened “read only?” We need to send a copy. “Write only?” No need to send anything. But the issue here is—there is no standard way to determine if a file was opened “read-only” or for “write-only” or even “read/write but writes are appended.” There are non-standard ways of doing it, but it varies from system to system.

Now, about that second field. It's a pointer to a C function, and as I stated before, serializing a Lua function in C is not possible. I worked around it by tagging the name of the function, but there's no name for this function, at least, one known to Lua. And why point to a hidden function to close the file? Because for files, you need to call the C function fclose(), but for a “file” opened by popen() (a function to run another process and either write data to it, or read data from it) you need to call the C function pclose(). That's why the function pointer.

And that's for a well known, standard Lua userdata. There are other types of userdata, depending upon the Lua modules in use, such as LPeg (where the full userdata for that represents an entirely different virtual machine geared for parsing text) or even my own org.conman.pollset (which is usually used to multiplex I/O with network connections, but can be used for other devices such as serial ports, and whose implementation depends on the operating system, as if things weren't difficult enough).

This also depends upon the module handing a given full userdata already loaded when deserializing. I'm not saying this is all impossible, but I do think it's beyond the scope of a general library to handle. Perhaps having the user register some type of callback to handle userdata might be required. I already support callbacks for tagged CBOR data, so adding support for full userdata using callbacks shouldn't be that hard.

All that's left—threads.

Thursday, April 02, 2015

Thank god that's over with!

At least yesterday is now over and those suffering from aphrilophobia can climb out from under their beds and go about their business for the next 365 days (next year being a leap year).

Serializing Lua data, part IV

The last Lua data type to serialize—threads.

It's more difficult than serializing a function, but probably a bit easier than serializing a userdata. Possibly. I haven't actually tried it yet, as it can't be done through Lua.

The issue is that there's no API to manipulate the callstack, and without that, you're stuck with dipping into the internals of Lua via C, which explains why most existing serialization modules haven't bothered with threads. And the only one that has attempted it, only works for Lua 5.1.

And that's why I didn't bother with it, either.

It was an interesting experience to write a serialization library for Lua. It became apparent exactly why so many don't bother with functions, userdata or threads—they're not easy to universally support.

I also skipped out on supporting metatables, not because they're hard, but because I just didn't get around to it. I also think that CBOR makes for a good serialization format. The primitives are well chosen, it's a consistent format and the semantic tagging makes it easy to extend.

And like I said earlier, I had fun playing around with this stuff.

Friday, April 03, 2015

A little snippet of permutations

I'm working on a little Lua project, and I found myself with a surprisingly hard problem. The task, take the following string:

one/{two three four five}/alpha/beta/{gamma delta epsilon}.c

and generate all possible permutations of the string with the words in the braces appearing once. In essence, generate the following list:

one/two/alpha/beta/gamma.c
one/two/alpha/beta/delta.c
one/two/alpha/beta/epsilon.c
one/three/alpha/beta/gamma.c
one/three/alpha/beta/delta.c
one/three/alpha/beta/epsilon.c
one/four/alpha/beta/gamma.c
one/four/alpha/beta/delta.c
one/four/alpha/beta/epsilon.c
one/five/alpha/beta/gamma.c
one/five/alpha/beta/delta.c
one/five/alpha/beta/epsilon.c

Parsing the string into a usable format was trivial (code left to the reader—hint: LPeg). So, starting from the output of parsing:

{
  "one/",
  {
    "two",
    "three",
    "four",
    "five"
  },
  "/alpha/beta/",
  {
    "gamma",
    "delta",
    "epsilon",
  },
  ".c"
}

Then … what?

You have to iterate through the main table, and at the same time, iterate through any subtables that might exist (anywhere from zero on up). It took me a while to come up with the code. I knew that there was an elegant way to do this, and by God, I was going to find it.

Several hours later, and:

-- ***************************************************************
--
-- Copyright 2015 by Sean Conner.  All Rights Reserved.
-- 
-- This library is free software; you can redistribute it and/or modify it
-- under the terms of the GNU Lesser General Public License as published by
-- the Free Software Foundation; either version 3 of the License, or (at your
-- option) any later version.
-- 
-- This library is distributed in the hope that it will be useful, but
-- WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
-- or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public
-- License for more details.
-- 
-- You should have received a copy of the GNU Lesser General Public License
-- along with this library; if not, see <http://www.gnu.org/licenses/>.
--
-- Comments, questions and criticisms can be sent to: sean@conman.org
--
-- ********************************************************************

-- ***********************************************************************
-- Lua 5.3 renamed the unpack() function to table.unpack().  So check for
-- Lua 5.3 and handle accordingly.
-- ***********************************************************************

if _VERSION == "Lua 5.3" then
  unpack = table.unpack
end

-- ***********************************************************************
-- This will take an array of strings and tables, and return an iterator
-- that will return successive permutations of strings.  
--
-- The table is unpacked into arguments, and the add() function will slowly
-- walk down the argument list, calling itself recursively to generate all
-- possible strings from the list of arguments and yield them one at a time.
--
-- The expand() function will create the coroutine and return a function
-- usable by the for keyword.
-- ***********************************************************************

function expand(list)
  local function add(a,x,...)

    -- ------------------------------------------------------------
    -- if x is nil, then we've exhausted all the paramters and have
    -- accumulated a string we can yield.  So yield it up.
    -- ------------------------------------------------------------

    if not x then
      coroutine.yield(a)

    -- --------------------------------------------------------------
    -- if x is a string, call ourselves with the contatenation of our
    -- accumulator string with x, and the rest of the paramters
    -- --------------------------------------------------------------
    
    elseif type(x) == 'string' then
      add(a .. x , ...)

    -- ----------------------------------------------------------------------
    -- otherwise, we have a table.  So iterate through the table, calling
    -- ourselves each time with an updated accumulator value and the rest of
    -- the parameters.
    -- ----------------------------------------------------------------------

    elseif type(x) == 'table' then
      for i = 1 , #x do
        add(a .. x[i], ...)
      end
    end
  end
  
  return function(co)
    local okay,res = coroutine.resume(co)
    return res
  end,coroutine.create(function() add("",unpack(list)) end)
end

-- ***********************************************************************
-- The parsing of 
--
-- 	one/{two three four five}/alpha/beta/{gamma delta epsilon}.c
--
-- into a table is left to the reader for an exercise.  For right now, I'm
-- using a hardcoded table.
-- ***********************************************************************

test = 
{
  "one/",
  {
    "two",
    "three",
    "four",
    "five"
  },
  "/alpha/beta/",
  {
    "gamma",
    "delta",
    "epsilon",
  },
  ".c"
}

-- ***********************************************************************
-- And now, just expand the list, printing each result.
-- ***********************************************************************

for path in expand(test) do
  print(path)
end

It's a decent example of recursion (where the trick is find the right base case and the reductions that lead to said base case). There might be non-recursive solutions, but I shudder at the potential complexity of such solutions.

Saturday, April 04, 2015

Some thoughts on make

From time to time, I find make limiting, think there has to be a better way and I start playing around with the idea of building a new make. It's not that uncommon for programmers to think this (SCons, CMake, ant, rake and a bunch of other programs) but oddly enough, as I try them, I keep going back to make (specifically, GNU make; note that when I mention make, I am talking about GNU make).

I think the main reason I go back is that make is mainly a syntax-light declarative langauge with bits of shell scripting thrown in. The other build systems are generally based around another language I do not know, they tend towards being very imperative, and rarely can you parallelize building the software (and when you get the makefile right, a parallel run of make just flies).

So when I had that “let's remake make” itch this time, I thought that perhaps I would see if I could stay within the confines of the existing syntax of make as much as possible. I wasn't trying to actually program my own make, but rather, just play around with how I would like make to look and work.

The main problem with make is declaring the dependencies. For instance, I'm embedding a UUID library into a larger project, and because recursive make is considered harmful, I include in my makefile:

lib/libspcuuid.a : build/third_party/uuid/luauuid.o	\
		build/third_party/uuid/uuid_ns_dns.o	\
		build/third_party/uuid/uuid_ns_null.o	\
		build/third_party/uuid/uuid_ns_oid.o	\
		build/third_party/uuid/uuid_ns_url.o	\
		build/third_party/uuid/uuid_ns_x500.o	\
		build/third_party/uuid/uuidlib_cmp.o	\
		build/third_party/uuid/uuidlib_parse.o	\
		build/third_party/uuid/uuidlib_toa.o	\
		build/third_party/uuid/uuidlib_v1.o	\
		build/third_party/uuid/uuidlib_v2.o	\
		build/third_party/uuid/uuidlib_v3.o	\
		build/third_party/uuid/uuidlib_v4.o	\
		build/third_party/uuid/uuidlib_v5.o

build/third_party/uuid/luauuid.o	: third_party/uuid/src/luauuid.c
build/third_party/uuid/uuid_ns_dns.o	: third_party/uuid/src/uuid_ns_dns.c
build/third_party/uuid/uuid_ns_null.o	: third_party/uuid/src/uuid_ns_null.c
build/third_party/uuid/uuid_ns_oid.o	: third_party/uuid/src/uuid_ns_oid.c
build/third_party/uuid/uuid_ns_url.o	: third_party/uuid/src/uuid_ns_url.c
build/third_party/uuid/uuid_ns_x500.o	: third_party/uuid/src/uuid_ns_x500.c
build/third_party/uuid/uuidlib_cmp.o	: third_party/uuid/src/uuidlib_cmp.c
build/third_party/uuid/uuidlib_parse.o	: third_party/uuid/src/uuidlib_parse.c
build/third_party/uuid/uuidlib_toa.o	: third_party/uuid/src/uuidlib_toa.c
build/third_party/uuid/uuidlib_v1.o	: third_party/uuid/src/uuidlib_v1.c
build/third_party/uuid/uuidlib_v2.o	: third_party/uuid/src/uuidlib_v2.c
build/third_party/uuid/uuidlib_v3.o	: third_party/uuid/src/uuidlib_v3.c
build/third_party/uuid/uuidlib_v4.o	: third_party/uuid/src/uuidlib_v4.c
build/third_party/uuid/uuidlib_v5.o	: third_party/uuid/src/uuidlib_v5.c

Or, if I wanted to save myself some typing:

define OBJECT_template = 
  $(1) : $(2)
endef

UUIDSRC = $(wildcard third_party/uuid/src/*.c)

lib/libspcuuid.a : $(subst third_party/uuid/src,build/third_party/uuid,$(UUICSRC))

$(foreach target,$(UUIDSRC),$(eval $(call OBJECT_template(build/third_party/uuid/$(notdir $(target)),$(target)))))

and have no idea what is going on three months from now (if indeed, I got that right). Instead, I would like to type:

lib/libspcuuid.a : build/third_party/uuid/*.o
build/third_party/uuid/*.o : third_party/uuid/src/*.c

and be done. This should be possible. “lib/libspcuuid.a” doesn't exist, but it depends upon all the “.o” files in “build/third_party/uuid”, which don't exist, but we have a rule that says “for ‘.o” files in ‘build/third_party/uuid’, there should be a corresponding ‘.c” file in ‘third_party/uuid/src/’ that is is generated from.” Which also leads to another problem with make: if “build/third_party/uuid” doesn't exist, make should make it! Automatically! I mean, make has no problems with “uuidlib_v5.o” not existing—that's the reason we're using make in the first place, to make files! If a directory a file lives in doesn't exist, make should make that too. Right?

By the same token, if I wanted to include some Lua modules but not all of them, I would like to do:

SPCMODC = env errno fsys hash math net pollset process strcore sys syslog
SPCMODL = debug getopt string table unix

lib/libspcmod.a : build/third_party/lua-conmanorg/*.o

build/third_party/lua-conmanorg/*.o : third_party/lua-conmanorg/src/{ $(SPCMODC) }.c \
	third_party/lua-conmanorg/lua/{ $(SPCMODL) }.lua

(Ah! So that's where yesterday's code snippet came from!)

I'll spare you the expansion.

Another small bit of annoyance is writing implicit rules to build the files. I learned the hard way that in make, this:

%.o : %.c
	$(CC) $(CFLAGS) -c -o $@ $<

only works if the “.c” and “.o” files are in the same directory! So the above rules works fine for:

long/path/to/src/foo.o : long/path/to/src/foo.c
src/bar.o              : src/bar.c
snafu.o                : snafu.c

But not for:

build/foo.o    : long/path/to/src/foo.c
build/bar.o    : src/bar.c
build/snafnu.o : snafu.c

Nope, I also have to write:

build/%.o : long/path/to/src/*.c
	$(CC) $(CFLAGS) -c -o $@ $<

build/%.o : src/%.c
	$(CC) $(CFLAGS) -c -o $@ $<

build/%.o : %.c
	$(CC) $(CFLAGS) -c -o $@ $<

if I want to segregate the “.o” files from the “.c” files.

How hard is it to realize that if I have a “.c” file, then the command to make a “.o” file is the same, regardless of where the “.c” file comes from, or where the “.o” file is being generated. Hello?

And while we're on that subject, one implicit rule I use is the following:

%.o : %.lua
	$(LUAC) -o $(@D)/$(*F).out $<
	$(BIN2C) -9 -o $(@D)/$(*F).c -t $(*F) $(@D)/$(*F).out
	$(CC) $(CFLAGS) -c -o $@ $(@D)/$(*F).c
	$(RM) $(@D)/$(*F).out $(@D)/$(*F).c

That bit of jibberish first compiles the Lua code using luac, then that output is converted to a C file which is then compiled into a object file so I can link it into the final executable, them removes the temporary files it needed to do all that. The noise you see is the cruft necessary to specify temporary files that won't get overwritten when doing a parallel build. Much nicer would be something like:

%.o : %.lua
	$(LUAC) -o $$.1 $<
	$(BIN2C) -9 -o $$.2 -t $(*F) $$.1
	$(CC) $(CFLAGS) -c -o $@ $$.2
	$(RM) $$.1 $$.2

(even better—make automatically deletes the intermediate files unless I tell it not to)

One complaint about make (or rather, makefiles themselves) is that a change to the makefile does not cause a recompile, mainly because the makefile itself is never listed as a dependency anywhere. This is never what you want! If the makefile was listed as a depenency (either explicitly or implicitly), then adding a new file to be compiled in would cause everything to be recompiled, when it should be the new file, any files that call code in the new file, and a final relinking of the code is all that is needed. Also, if you change a variable, like $(LDFLAGS), then all that should happen is any rule that uses $(LDFLAGS) should be run, not a complete build.

In other words, you really have a dependency on a portion of the makefile, not the makefile as a whole. But solving this would require a possible rethink of how make works (perhaps a cached version of the makefile that mode checks against, and intelligently applies the changes as dependencies? You know, so if you change $(CC), any rule using $(CC) is automatically run).

I realize these some of my ideas are not that easy to implement (heck, I wasted a few hours yesterday writing code to properly build a list of targets and dependencies based on ideas presented here), but I think they may go a long way to making make less horrendous to use.

Sunday, April 05, 2015

No Reservations in Narnia this fine Ēostre day

Ah, today's holiday. A celebration of the reanimation of a Jewish carpenter using Nordic symbols, named after a Germanic deity and involving sweets derived from Central American food. Ain't modern life wonderful?

As for me, I'm spending the day reading about Anthony Bourdain's eating his way through Narnia.

Monday, April 06, 2015

Someone bought Casa New Jersey and you have to see what happens next

When I left Casa New Jersey, it needed quite a bit of loving care. But sure enough, someone came along and lavished quite a bit of loving care on the house. And seriously, what a transformation! (link via Spring on GoogleMyFacePlusSpaceBook).

I especially like the work done upstairs. Wow!

Tuesday, April 07, 2015

Just use /dev/urandom and be done with it

I noticed one of the developers at The Ft. Lauderdale Office of The Corporation using the time of day to seed a random number generator, which is borderline okay (depending on how the resulting random numbers will be used) there are better ways to generate a random seed, at least on a modern POSIX system—read data from /dev/urandom.

My fellow cow-orker B, with whom I was having this discussion, mentioned this borderline paranoid approach to reading /dev/urandom. But I think that if you have to call fstat() to make sure the file is actually /dev/urandom then you have more things to worry about (really—if a cracker can substitute /dev/urandom with known data, it's pretty much game over—B agreed with that statement, by the way). Besides, the author wasn't paranoid enough! Who's to say there isn't some extra code in there (say, via $LD_PRELOAD or ptrace() or maybe even through some ELF magic on the executable) that intercepts the read() function to return “random data” when reading from /dev/urandom? Hmmmm? (about the only thing you can do to counter that is nuke the site from orbit—it's the only way to be sure)

But in the mean time, just use /dev/urandom.

Wednesday, April 08, 2015

Privacy? At GoogleMyFacePlusSpaceBook? Surely you jest!

I was inspecting Facebook's network traffic today in Firefox Devtools, when I realized that any text I put into the status update box was sent to Facebook's servers, even if I did not click the post button. Ever curious, I Googled this behaviour and came across a study which reveals some very frightening information:

Facebook calls these unposted thoughts "self-censorship," and insights into how it collects these nonposts can be found in a recent paper written by two Facebookers.

Via Lobsters, Facebook sending 'nonposts' to its servers and storing unpublished thoughts

I wouldn't be surprised if all the sites in the MyFaceGoogleBookPlusSpaceosphere aren't doing the same thing. And the thing is, when I do a search on “experimenting on unwilling subjects” the results all seem to be Na zis Nazis Nazis for some reason. Just approach FaceGoogleMyPlusSpaceBook that everything posted, whether “private” or not, is going to be seen by everybody. Because it is.

Thursday, April 09, 2015

Harry Potter and the Methods of Rationality

Finally!

Harry Potter and the Methods of Rationality is finally finished! I'm so happy!

I started reading this a few years ago, but the rate of new chapters was never that fast and it's only been in the past month or so that Eliezer Yudkowsky finished writing it. I think the premise is wonderful—that Harry Potter was raised by scientists and skeptically approached magic at Hogwarts by applying rational thought and scientific experimentation. It takes a while to start, but once it does, I found I could not stop reading it (well, until I reached the current chapter and had to wait a several months for the next outburst of chapters).

Over the past two nights, I've finished reading it (staying up way past my bedtime), and I must say, I found He-Who-Shall-Not-Be-Named to be a quite facinating character (when he finally did show up—oh, is that a spoiler?).

I think it's worth the read, all 122 chapters of it. Don't worry, it's a complete story with Harry winning and He-Who-Shall-Not-Be-Named losing and not a seven book series, so don't let the length discourage you if you are a Harry Potter fan—at least now you can finish it in one reading binge if you so desire.

Friday, April 10, 2015

Notes about a conversation overhead in the kitchen at Chez Boca

“Aaaaaaaaaah!”

“What happened?”

“The 16 ounce bottle of vanilla extract fell on the floor!”

“Did the bottle break?”

“No, but there's vanillia extract all over the pantry.”

“Was it the used bottle?”

“It's used now.”

“Is there any left?”

“About a quarter of the bottle is left.”

“Sigh. That was the good stuff.”

“At the very least, it will smell incredible in here for a while.”

Saturday, April 11, 2015

Coyote's Flying Saucer Retrievals and Repairs

It's nice to know that Coyote's Flying Saucer Retrievals and Repairs exists so when my C-57D needs repair, I know where to go to have it checked out. The only bad thing is that it is in California and not in Roswell or Rachel where all the flying saucer activities seem to happen in this country. I guess I'll have to keep in mind that if I crash my C-57D, I need to crash in or near Ocotillo, California.

Sunday, April 12, 2015

For me, SPF is not that great an anti-spam measure

I was reminded of SPF the other day. It's an anti-spam measure, primarily to help identify email spoofing. I set up an SPF record on my domain years ago, but other than that, I haven't really done anything else with it. But being reminded of it, I thought it might be a good idea to see just how effective it could be. I'm already using a greylist daemon to cut down on spam but hey, the more spam that is caught, the better.

First step—just how effective is the greylist daemon? I have the logs from the greylist daemon for the past month (March 15^th to April 11^th). Some processing on the logs and I have my answer:

Unique emails processed by the greylist daemon
Emails accepted	5,028
Emails rejected	5,132
Total	10,160

Wow!

A bit over 50% of the emails received by my email server are spam. I'm not sure if I should be depressed that easily half the email addressed to me is spam, or happy that the greylist daemon is an easy way to avoid false positives. I suppose both are in order.

Now, I still get spam despite the greylist daemon, but all that means is that the sender is actually bothering to follow the SMTP protocol—not that high a bar. So, of the emails that do get through, how much would get flagged with SPF? Okay, time to check up on the SPF specification, and boy, is that a mess. The grammar to parse SPF records requires backtracking (lovely!—and lest you think a message from 2010 has any relevence to a 2014 standard, think again; the grammar didn't change all that much) and not entirely context free either (sigh—one letter in the macro-expansion has two meanings depending on where it appears).

Oh, and that grammar? It's actually covers three different grammars—one for parsing the SPF record itself, a second one to parse an email header, and the third a secondary text string via a secondary DNS query (the SPF record itself is obtained via a DNS query, by the way).

Okay, so munging the grammar to what I think is intended and leaving out what I don't need, I went through the log file and for each accepted email, did an SPF check according to the specification. Granted, the data I get now might not reflect the results made at the original time, but it should give me a baseline to go by.

For the test, I pulled out all emails accepted (5,028) and removed those I explicitly allowed (for example, accept anything from a given IP address, or from a given domain) or that did not have a sender address (allowed by the SMTP protocol to prevent mailing loops when generating mail bounce messages), leaving me with 4,343 emails. Then, for those, I looked up the SPF record for the given domain, and if it had one, applied its policy.

The 4,343 accepted emails came from 1,000 unique domains, of which only 433 had an SPF record. Okay, 43% of the domains have an SPF record. And of the domains that had an SPF record, only 629 emails accepted were checked. Or 12½% of all accepted incoming emails could be checked via SPF. Sigh.

But of those that were checked via SPF, how did we fare? Were a lot spam? Or were most acceptable forms of email pef SPF policy?

Results of applying SPF policy against incoming email
fail	43	IP address was not allowed to send this email
softfail	53	IP address should not be sending this email (used for testing)
neutral	90	IP address has no policy
pass	443	IP address is allowed to send this email

A 70% pass rate for SPF. Only 43, or almost 1% (or around two per day) could have been deleted as spam. Another 53 maybe, possibly, could have been deleted as spam. And 90 no idea one way or the other. Sigh.

You want to know what has a better rate of catching spam than SPF for my email? Any email addressed to my domain registration email not from the registrar. For me, I don't think it worth the effort to implement this.

Monday, April 13, 2015

One of the most impressive demoscene videos I've ever seen

This is simply amazing.

[Don't attempt this on an emulator. This will only work on a real, honest to God, IBM 5150 with an IBM CGA card. That shouldn't be hard to get, right?]

I'm impressed. I didn't think it was possible to get more than 16 colors from the CGA card and here is a demo showing over 1000 colors. On a stock IBM 5150 PC from 1981 (video via 8088 MPH: We Break All Your Emulators « Oldskooler Ramblings, from a discussion at Hacker News). And for those of you who are curious as to how this was done, there're indepth articles about the graphics programming (via Hacker News) and the sound programming (again, via Hacker News).

Tuesday, April 14, 2015

Honey on tap

The most successful fully funded crowdfunding campaign on Indiegogo is not for a new smartwatch, video game, or 3D printer. It is for a new way to harvest honey, a potential breakthrough in a practice that has not seen a significant technological advance since 1852, when the Rev. Lorenzo Langstroth patented America's first movable frame beehive.

The Flow hive has amassed $8.9 million from more than 25,000 backers in one month (the goal was just $70,000), a possible sign that the urban-hipster revival of beekeeping is still alive, even as the U.S. honeybee population continues to die off. (The USDA has sounded the alarm, estimating that a third of all honeybees have died since 2006. The main suspect is a class of neuro-active insecticides called Neonicotinoids.)

Via Instapundit, This 'Honey on Tap' Beehive Design Just Raised $9 Million on Indiegogo

I've heard of this, but this is the first time I've seen how the Flow™ Hive works, and it's pretty ingenious. I know T, my fellow cow-orker, might be interested in this as he keeps bees as a hobby.

Wednesday, April 15, 2015

The taxman cometh!

Telonephobics beware! For it is the Ides of April and the second inevitability of life is upon us. But fear not if you lack health insurance and cannot afford $95 or 1% of your income (whatever is higher) since it is very easy to avoid paying the ~~penalty~~ tax:

#4. Get beaten by your wife. Required documentation: None
Official text reads: "You recently experienced domestic violence. Required documentation: None."
Translation: Get attacked by your wife. You may then apply for the exemption by simply stating “I am exempt because my wife beat me.” No documentation required.

Via Captain Capitalism, Citizen Liberty: 17 Ways To Avoid The Obamacare Tax Penalty, Including Being Beaten By Your Wife

It doesn't surprise me that a 2,700 page bill that no Congresscritter read and over 33,000 pages of regulations (wow! Something longer than Atlas Shrugged) would have egregious loopholes in it.

Really?

NO DOCUMENTATION?

Sheesh.

Thursday, April 16, 2015

How do I know you're allowed to send me this?

I was pulled into an improptu design meeting at work. Originally it was to discuss the format of a new URI for our Android application to use, but it quickly shifted into an authentication issue on the Android platform.

I don't work on Android applications (I do the call-processing on the telephony network side, not the cellphone side), so my terminology might be a bit off but the gist of the issue is our application, named Awesome Application (name changed to protect me), comes preinstalled on Android phones. We've allowed another preinstalled application, Bodacious Bronies (name completely made up), not written by us, to, when a certain action is done by the user in Bodacious Bronies, launches our application. This is done by Awesome Application listening for an intent sent by Bodacious Bronies, and then doing it's thing.

But the issue the our developer, D, had with this is to prevent the Malevolent Malcontent application (for example) from spamming Awesome Application with repeated intents. The fear here is that Malevolent Malcontent could so annoy the user with our program always popping up that the user would then uninstall Awesome Application, or worse, bitch, complain and moan to the phone carrier to remove or disable our application post-haste.

Our application could check the uid or the package name of the intent sender and only do the thing it does if the sending application is allowed, but neither the uid or the package name is fixed; either one can change with an update, and if our application isn't updated with the new uid or package name, then our app does nothing since it doesn't know the intent is from an allowed application.

And other methods, like having Bodacious Bronies sign the intent (somehow) is still subject to attacks; in order to sign the intent, a private key needs to be stored with Bodacious Bronies, (and we were sure that the creators of Bodacious Bronies would not want to include a private key with the application) and what's to stop the creators of Malevolent Malcontent from nabbing that private key (the developer of Malevolent Malcontent could get an Android phone, jailbreak it, and extract the Bodacious Bronies private key) and forge intents?

Security is hard. So is authentication.

There is a way to keep intents from being broadcast to every application. D was trying to find a way to avoid this, as one of our customers wanted the broadcast method of intents (for some reason—again, I'm not an Android developer so I'm not sure what the trade-offs are here) but he decided that the best course of action is to use the non-broadcast intent method. Now he has to convince the Powers-That-Be that this is the only way.

Friday, April 17, 2015

Minnesota's great umlaut war is over

Today, we bring you urgent and breaking news out of Minnesota, where a battle over umlauts has been — well, not raging. What is the more polite version of raging? Occurring? Happening? Gently taking place? Something like that.

Anyway! Minnesota. Umlauts. See, there is a city in Minnesota that had been known as Lindström — or, if you saw the signs greeting you on the way in or out of town in recent years, Lindstrom.

Via Brian Yoder on MyFaceGoogleBookPlusSpace, Minnesota’s great umlaut war is over (also, Minnesota was having an umlaut war) - The Washington Post

My first thought was couldn't the MDOT just spell it “Lindstroem?” But then I read that Lindström has a sister city in Sweden, Tingsryd, and I wasn't sure if the umlaut served the same function in Swedish as it did in German. It turns out it doesn't, and the “ö” in Swedish is a distinct character, unlike in German where the “ö” is a shorthand notation for “oe.”

It all turned out fine though, the MDOT is going around adding umlauts on all the Lindström signs.

Ms. L'Engle also never mentioned the 24-hour news cycle that helped subjugate the population of Camazotz

Via a link on FaceGoogleMyBookPlusSpace is an article about a cut passage from an early draft of A Wrinkle In Time. The article talks briefly about the cut passage and then goes into some details about Madeleine L’Engle, but I can't help but quote from the cut passage:

So she said, “But Father, what's wrong with security? Everybody likes to be all cosy and safe.”

“Yes,” Mr. Murry said, grimly. “Security is a most seductive thing.”

“Well—but I want to be secure, Father. I hate feeling insecure.”

“But ~~not enough~~ you don't love security enough so that you guide your life by it, Meg. You weren't thinking of security when you came to resuce me with Mrs Who, Mrs Whatsit, and Mrs Which.”

“But that didn't have anything to do with me,” Meg protested. “I wasn't being brave or anything. They just took me.”

Calvin, walking beside them with his load of wood, said, smiling warmly at Meg, “Yes, but when we got here you didn't go around whining or asking to go home where you could be all safe and cosy. You kept yelling, where's Father, take me to Father: You never gave a thought to security.”

“Oh,” Meg said. “Oh.” She brooded for another moment. “But I still don't see why security isn't a good thing. Why, Father?”

“I've come to the conclusion,” Mr. Murry said slowly, “that it's the greatest evil there is. Suppose your great great grandmother, and all those like her, had worried about security? They'd never have gone across the ~~country~~ land in flimsy covered wagons. Our country has been greatest when it has been most insecure. This ~~longin~~ sick longing for security is a dangerous thing, Meg, as insidious as the strontium 90 from our nuclear explosions that worried you so about Charles Wallace when you read in science at school that it was being found in greater and greater quantities in milk. You can't see strontium 90. You can't feel it or touch it. But it's there. So is the panicky searching for conformity, for security. Maybe it's because of the Black Thing, Meg. Maybe this lust for security is like a disease germ that it has let loose on our land. I don't know, Meg. All I realize now is that my fight is much bigger than this little one on Camazotz.”

Despite being written over fifty years ago, it seems to apply more to us today than it did in 1962 (and here's a discription of Camazotz if you are unfamiliar with the book).

Saturday, April 18, 2015

Sadly, yet accurately, still no flying cars

Tell me, does any of this sound familiar?

(1) Random acts of violence by crazy individuals, often taking place at schools …

(2) The other major source of instability and violence comes from terrorists, who are now a major threat to U.S. interests, and even manage to attack buildings within the United States.

(3) Prices have increased sixfold between 1960 and 2010 because of inflation. …

(4) The most powerful U.S. rival is no longer the Soviet Union, but China. However, much of the competition between the U.S. and Asia is played out in economics, trade, and technology instead of overt warfare.

(5) Europeans have formed a union of nations to improve their economic prospects and influence on world affairs. In international issues, Britain tends to side with the U.S., but other countries in Europe are often critical of U.S. initiatives.

(6) Africa still trails far behind the rest of the world in economic development, and Israel remains the epicenter of tensions in the Middle East.

(7) Although some people still get married, many in the younger generation now prefer short-term hookups without long-term commitment.

(8) Gay and bisexual lifestyles have gone mainstream, and pharmaceuticals to improve sexual performance are widely used (and even advertised in the media).

(9) Many decades of affirmative action have brought blacks into positions of power, but racial tensions still simmer throughout society.

(10) Motor vehicles increasingly run on electric fuel cells. …

(11) Yet Detroit has not prospered, and is almost a ghost town because of all the shuttered factories. However. a new kind of music … has sprung up in the city.

(12) TV news channels have now gone global via satellite.

(13) TiVo-type systems allow people to view TV programs according to their own schedule.

(14) Inflight entertainment systems on planes now include video programs and news accessible on individual screens at each seat.

(15) People rely on avatars to represent themselves on video screens …

(16) Computer documents are generated with laser printers.

(17) A social and political backlash has marginalized tobacco, but marijuana has been decriminalized.

Oh, and let's not forget President Obomi.

Wait—what?

What you read was eighteen predictions (link via Hacker News) made by John Brunner is his 1969 novel Stand on Zanzibar. It's an incredible list, scarily accurate in its portrayal of life in 2010. I never read that book, but I did read Shockwave Rider that predicted a global network besieged with malware and The Sheep Look Up, a book about global environmental collapse that was the single most scary book I've ever read (that I try not to think about too much least I start having nightmares again). Both of those were very good (even if The Sheep Look Up is too horrifying to think about), so I would think Stand on Zanzibar would be great as well.

Sunday, April 19, 2015

The man behind the bits

This self-portrait:

[I swear I thought it was a red jelly bean!]

also happens to be a valid QR code. Go head, try it if you can. See how deep the rabbit hole goes.

For the curious, I used QArt Coder to generate the image. And for the really curious, the theory behind how it works.

Monday, April 20, 2015

Just because it was that type of day

[I don't think it has the whole toilet paper thing down quite yet]

Because a picture of a bunny with a pancake on its head is cliché.

Tuesday, April 21, 2015

Notes on an overheard conversation in The Ft. Lauderdale Office of The Corporation

“Do you know how I get to the timesheet application?”

“Timesheet application? You want to apply for some vacation time?”

“Yes.”

“Okay, first you need to log into the VPN.”

“But we're in the office, why do I need to log into the VPN?”

“No, you're thinkging of The Corpration VPN. You need to log into the Corporation Overlord Corporation VPN.”

“Oh. Where do I go to log into that?”

“Here, let me email you the location.”

“Thank you. Hmm … Ah, I use that account name … but I don't seem to know the password for that account. Is it the same as any other password you use?”

“No, it's a different password.”

“Sigh. … Okay it's not that password … and it's not that password. I'm afraid of trying it a third time lest I get locked out.”

“So once you reset your password and can log on, do not select the ‘Timesheet Application’ but instead select the ‘Monopolistic Database Corporation Application Suite’ instead.”

“Do tell.”

“Yeah, I can't make this stuff up even if I wanted to.”

“So skip ‘Timesheet Application’ and instead use the ‘Monopolistic Database Corporation Application Suite.’”

“Yes.”

“I won't ask.”

“Good, beacuse I don't know the answer.”

Notes on another overheard phonecall in The Ft. Lauderdale Office of The Corporation

“Okay, I copied down the new password.”

…

“Wait? I'm stuck with that password?”

…

“Oh, I can't change the password for ten days. Then I can change it to something less obvious.”

…

“Okay, fine. Now let me try logging into the site.”

…

“No, I don't have Internet Explorer.”

…

“No, I'm not running Windows 7, or any version of Windows for that matter.”

…

“A Macintosh.”

…

“No, really.”

…

“No, I'm not running Google Chrome.”

…

“Firefox.”

…

“Is there a joke I didn't get?”

Wednesday, April 22, 2015

Surely you can't be serious

A couple of months ago, I was at a party somewhere, and a boy came up to me who was, like, 8 or 10 years old, and he said, “Oh, I really liked Airplane! I thought it was really funny!” And I said, “How was it that you came to see it?” And he said, “Well, my grandfather made me watch it.” [Laughs.] If you’d told us in 1980 that the grandkids of the audience would be the ones who’d keep the movie going, it would’ve been very gratifying. But I don’t think we ever anticipated it. And it’s one of the great thrills, I think, of all of our lives that it still remains well known.

Via Instapundit, Surely you can’t be serious: An oral history of Airplane! · Oral History · The A.V. Club

It's an oral history of the making of “Airplane!” from many people who were involved in the making of one of the funniest movies to ever come out of Hollywood. And stop calling me Shirley.

Thursday, April 23, 2015

Gotta love spam with a legal disclaimer

From

"John" <yjonjens@mail.com>

To

sean@conman.org

Subject

business leads

Date

Thu, 23 Apr 2015 17:22:50 +0200

Hey,

You are receiving this email because we wish you to use our email marketing service.

We wish to be your email marketing partner, we can grow your business 2-5 times than now.

If you would require more information please send us an email and we would be glad to discuss the project requirements with you soon. Looking forward to your positive response.

Kind Regards
John
Email: pottleyo@aliyun.com

This e-mail message and its attachments (if any) are intended solely for the use of the addressee(s) hereof. In addition, this message and the attachments (if any) may contain information that is confidential, privileged and exempt from disclosure under applicable law. If you are not the intended recipient of this message, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. If you have received this message in error, please promptly notify the sender and immediately delete this message from your system.

If you don't wish our future news letter, pls send address to ttickmay@aliyun.com for removal.

I'm only reproducing this because of the disclaimer. Really? You're fishing for clients, not giving legal, medical or confidential information. It doesn't mean a thing. Also, when I tried searching for this, Google helpfully mentioned:

Did you mean: This e-mail message and its attachments (if any) are intended solely for the use of the addressee(s) thereof. In addition, this message and the attachments (if any) may contain information that is confidential, privileged and exempt from disclosure under ap…

Heh.

Also, that text shows up on a lot of emails. A lot.

I would like to note that this came from 135328.com, a domain registered in China, from a server in Williamsville, NY. The email was from yjonjens@mail.com, which is from a domain registered to a company in Chesterbrook, PA and administered by a German company in Karlsruhe. The default Reply-To: address is broling@aliyun.com, which is from a copmany in China registered by what appears to be either a European or American. And as you can see, it doesn't match the “sender” address, nor the address mentioned in the email itself. I'm not worried about being sued by these jokers.

Friday, April 24, 2015

Notes about a Lua lint checker

Luacheck is a static analyzer and a linter for Lua. Luacheck detects various issues such as usage of undefined global variables, unused variables and values, accessing uninitialized variables, unreachable code and more.

Luacheck

The one real issue I have with Lua is its dynamic typing. Of all the bugs I fix in my own Lua code, I would say that the majority are due to typos (wrong variable name) or an unexpected type. So I was quite happy to come across and try out Luacheck. And fortunately, it's pretty straightforward to run.

I ran it over “Project: Sippy- Cup” and … wow. The extensive regression test I have has already flushed out the typos and the unexpected type errors I tend to make. But Luacheck found quite a few unused variables (which is nice—it also found a bunch of unsused LPeg expressions) and a ton of unintentional global variables (because I forgot to declare them with local).

The output is easy to read (here's a representative sample from some non- work related code I have):

Checking ptest-cr-select.lua                      Failure

    ptest-cr-select.lua:53:9: variable amount was previously defined as an argument on line 52
    ptest-cr-select.lua:128:9: variable okay is never accessed
    ptest-cr-select.lua:193:40: unused argument event
    ptest-cr-select.lua:197:43: shadowing upvalue conn on line 194
    ptest-cr-select.lua:213:21: shadowing upvalue argument event on line 193
    ptest-cr-select.lua:215:15: unused variable rem
    ptest-cr-select.lua:215:15: shadowing upvalue rem on line 194

Total: 7 warnings / 0 errors in 1 file

About the only false positive it finds is this idiom:

function foo(param1,param2)
  local param1 = param1 or "default value"
  local param2 = param2 or 3
  local a = ...
  -- ...
end

where it will flag param1 and param2 as shadowing an upvalue. This idiom though, is used to provide a default value if a parameter isn't given to a function. It's easy enough to fix, either:

function foo(param1,param2)
  param1 = param1 or "default value"
  param2 = param2 or 3
  local a = ...
  -- ...
end

function foo(param1,param2)
  local param1 = param1 or "default value" -- luacheck: ignore
  local param2 = param2 or 3 -- luacheck: ignore
  local a = ...
  -- ...
end

Overall, I'm glad I found this tool. It's been a real eye opener.

Saturday, April 25, 2015

Jamie and Adam Unleased.

Bunny and I just saw Jamie & Adam UNLEASHED at the Kravis Center. What a fun show. Adam and Jamie would select ~~crash test dummies~~ ~~victims~~ volunteers (and there were no shortage of thoses) to come up on stage to help demonstrate some principle of physics, such as a nine year old girl lifting two grown men a foot above the stage, or arranging four men in a sitting position without chairs, and other physics-based tricks.

They also talked about several myths they've done on their show Mythbusters which included lots of explosions. A lot of explosions. Including a several minute clip of various things they've exploded over the years (like water heaters, cars, buildings, cement trunks, more cars and in general, nearly every type of explosive device you can conceive of) that nearly brought down the house (literally, since they boosted the base so you could feel the explosions rattling the theater).

And to end the show, they shot a volunteer with a paintball gatling gun for what seemed like a solid minute (don't worry—the volunteer was wearing the suit of armor Adam wore to protect him underwater from sharks) leaving one paint covered volunteer and a volunteer shaped space on the wall behind him.

Very amusing stuff.

But I think the most amusing thing to happen at the show happened during intermission. I received the following text message:

[“Is this still Sean Conner's phone?” “Yes” “Look to your right”]

I didn't recognize the number, and it took me a few moments to decide to even answer “Yes.” The response to my response was classic, and indeed, when I looked to my right, I saw my old roommate Rob, his wife Laura, Squeaky and his wife Tanya, sitting at the other end of the aisle.

'Tis a small world indeed.

Sunday, April 26, 2015

Just a quick note on something I noticed going on today on the home network

The script kiddies are active tonight.

Chain ssh-block (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   17  1812 REJECT     all  --  *      *       188.135.202.39       0.0.0.0/0           reject-with icmp-port-unreachable 
   38  2272 REJECT     all  --  *      *       113.106.85.23        0.0.0.0/0           reject-with icmp-port-unreachable 
    4   348 REJECT     all  --  *      *       117.253.105.235      0.0.0.0/0           reject-with icmp-port-unreachable 
   19  2080 REJECT     all  --  *      *       37.190.87.219        0.0.0.0/0           reject-with icmp-port-unreachable 
   20  2316 REJECT     all  --  *      *       187.72.49.52         0.0.0.0/0           reject-with icmp-port-unreachable 
   16  1796 REJECT     all  --  *      *       201.75.109.180       0.0.0.0/0           reject-with icmp-port-unreachable 
  512 25388 REJECT     all  --  *      *       218.83.6.81          0.0.0.0/0           reject-with icmp-port-unreachable 
   20  2248 REJECT     all  --  *      *       177.70.122.255       0.0.0.0/0           reject-with icmp-port-unreachable 
   15  1800 REJECT     all  --  *      *       117.253.215.122      0.0.0.0/0           reject-with icmp-port-unreachable 
   17  2032 REJECT     all  --  *      *       117.244.25.226       0.0.0.0/0           reject-with icmp-port-unreachable 
   18  2048 REJECT     all  --  *      *       134.255.165.240      0.0.0.0/0           reject-with icmp-port-unreachable 
   17  1964 REJECT     all  --  *      *       187.49.248.42        0.0.0.0/0           reject-with icmp-port-unreachable

These are just the script kiddies caught trying to brute force a login to my home machine over the past hour (they're blocked after five attempts, and the block remains for an hour lest I end up with hundreds of entries). I wonder if there's a quota they have to meet?

Monday, April 27, 2015

A shot to the arm

I sit down. As soon as I do, the phlebotomist drops a padded bar across the seat. Sure, they claim its for resting your arm while they draw blood, but it's real purpose is to keep people from escaping. Most people don't hear the soft click as it's locked into position.

Then my arm is wrapped with a large rubber band, cutting circulation to my hand. Again, the claimed reason is to help a vein rise to the surface of the arm, but in reality, it's there to weaken the arm so you can't fight.

I'm surprised that I'm letting them do this to me. There must be something in the air to keep me docile throughout the procedure.

“This won't hurt a bit.” A lie. Not quite as bad as “the check is in the mail,” or “I'm from the government, I'm here to help,” but it's still a lie.

“Aaaaaaaaaaaaaaaaaiiiiiiiiiiiiiiiiieeeeeeeeeeaaaaaarrrrrrrrrrrrrrg!”

“I haven't started yet.”

“Just pract—aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaah!”

“Just a few moments more.”

“Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhhhh!”

“Be patient.”

“Errrrrrrrrrrrrr—are you a phlebotomist, or a vampire?”

“That remark cost you three more vials.”

“Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaarrrrrrrrrrrreeeeeeiiiiiiii!”

“There we go, that wasn't so bad, was it?”

“Aaaaaaaaaaaaaaaaaaaaaaaaaah!”

“The needle is out.”

“Aaaaaaaaaaaa—oh.”

“Here, hold this cotton pad here.”

I reach over with my other, non-blood-starved hand and place it over the cotton pad to staunch the flow of blood. The phlebotomist then tapes the cotton pad and my hand to my arm. “Um, you taped my hand to my arm.”

A very subtle click as the bar across the chair is lifted. “You are free to go.”

“Um. My hand is still taped to my arm.”

“Just pull it off. It won't hurt.”

Tuesday, April 28, 2015

Breaking Brain

Everybody knows how to ride a bike, right? But not everybody unlearns how to ride a bike. But Destin Sandlin did just that (link via Jason Kottke and no, it did not involve injury or a rare disease—he did this intentionally). He also untaught his six-year old son, who proved more adaptable because he was younger.

So, how do you unlearn riding a bike? Well, first off, you get a bicycle that steers backwards …

Wednesday, April 29, 2015

Alien packets from outer space

To help test “Project: Sippy-Cup,” I wrote a mock component to return precanned data releated to our test data. The mock component is basically a specialized DNS server that only expects a certain type of query. I wrote it so it would be easier to configure and run than a full blown bind installation.

But for the past few weeks it was crashing and pretty much the only report I would get back is “it crashed.” Sigh. I did find the error path in the mock component to be a bit spotty, but this was never intended to be a full blown product but rather something that would work just well enough to get the testing done. I never tested it with bogus queries because I never expected it to get bogus queries. We had control over all the data. The mock component would only talk to programs we were testing.

Even after fixing some “how did I not see that error?” type problems, the mock component was still crashing, and the only way that could happen is if the queries being sent were too large (over 512 bytes), the query was corrupted or malformed and could not be decoded, or if the query wasn't the single query type supported by the mock component. And there was no way any of that could happen. We controlled all the data!

Or so I thought.

It turns out the mock component was receiving God knows what from random computers on the Internet, which is incredible when the computer the mock component is running on doesn't have a public IP address!

Well, okay, it does have a public address, but it's a public IPv6 address, but the queries causing the crashes were all coming from IPv4 addresses.

Wow.

About the only thing I got to explain that behavior is the IPv6 address is routed via a tunnel, and perhaps there's some routing leakage that lets public IPv4 packets through. Other than that, I got nothing.

At the very least, I did fix the dodgy error handling so the mock component doesn't crash from data that it shouldn't get.

A trip to the hospital

Bunny was in the process of cutting up an old computer desk on the back porch with a sawzall when a chunk fell off in such a way not to break her big toe, but rip the nail off of it (ugh).

[It is perhaps a bad idea to cut heavy objects while not wearing steel tipped shoes.]

She managed to bandage it up and off we headed to the non-emergency emergency room at the local hospital. Bunny said it was more of a clinic than an emergency room but that it happens to be located in the emergency room.

Ooookay.

I drop her off, then spent the next ten minutes trying to locate the parking lot. Turns out that at the Boca Raton Regional Hospital, the emergency room parking lot (as well as the non-emergency emergency room parking lot) was across the street, marked by a few tiny signs with a low albedo.

Oddly enough, the emergency room wasn't crowded, and thus non-emergency emerency room wasn't crowded. By the time I had arrived in person, Bunny was already in the bowels of the hospital being treated.

And I'm stuck in the waiting room watching “Jurassic Park III.”

Limping, but at home

A bunch of people got eaten. Sam Neil managed to escape being eaten. The dinosaurs still roam that Central American Island like they own the place or something. And Bunny finally limped out of the non-emergency emergency room.

[And here we see the very latest in the Spring Line of Foot Braces.]

One clunky foot brace, some good medicine, dinner from Denny's (slogan: “You never start out for us, but you'll always end up here”) and all is, mostly, right with the world.

Thursday, April 30, 2015

Some perils of handing signals in Lua

This is one of those “Oh, yeah, I didn't think that through, did I?” type of bug.

I wrote a signal module for Lua, which can handle both ANSI C and POSIX signals with largly the same API (the POSIX implementation one has some additional functions defined).

Handling signals in Lua is not that straightforward because of the nature of signals—you are effectively writting multithreaded code. You just can't call back into Lua from the signal handler (while the Lua VM has no static data and each Lua state is isolated unto itself, two threads sharing a Lua state can lead to problemss). The only Lua function you can safely call is lua_sethook(), which can be used to stop the Lua VM at the next VM instruction (it's typically used for debugging and signal handing). This callback can then call back into Lua. It is a bit convoluted (the signal handler will call lua_sethook() and return; the Lua VM will resume and then call the hook), but it does allow you to write signal handlers in Lua:

signal.catch('windowchange',function()
  print("Wheeee!  Our terminal just resized!")
end)

and not have it blow up on you.

So, with that in mind, I give you this code:

local net    = require "org.conman.net"
local clock  = require "org.conman.clock"
local signal = require "org.conman.signal"

local raddr = net.address("127.0.0.1",udp,'echo')
local sock  = net.socket(raddr.family,'udp')

signal.catch('alarm',function()
  sock:send(raddr,tostring(clock.get()))
end)

clock.itimer(1)

local previous = clock.get()

while true do
  local _,data = sock:recv()
  local now = clock.get()
  if data then
    local zen = tonumber(data)
    print(string.format("%.7f\t%.7f",now - zen,now - previous))
    previous = now
  end
end

This is a UDP echo client program. signal.catch() handles the alarm signal (SIGALRM) by sending a packet of data (which is just the current time) to the echo server. clock.itimer() informs the kernel to send the alarm signal once a second. So once a second, our program receives the alarm signal and sends the current time. Then, in an infinite loop, we just wait for packets to arrive (which should be the packets we sent to the echo server—they're “echoed” back to us) and we calculate how long the packet took round trip and how long it was from the previous packet. The output looks like:

0.0002971	1.0014961
0.0003922	0.9999950
0.0002851	0.9998930
0.0003171	1.0000319
0.0003910	0.9999740
0.0002551	0.9998641
0.0003359	1.0000808

The first column is the round trip time (in seconds) for the packet (around 3 to 4 ten thousandths of a second), and the second column is how long (in seconds) from the previous packet (a second, give or take a few ten thousandths).

But our call to sock:recv() is interrupted by the alarm signal. Unfortunately, one side effect of signals is that they will interrupt “long running” system calls, which is almost always system calls dealing with I/O, such as read() or write(). When such a call is interrupted, the system call will return an error of EINTR. We can see this if we change the code a bit:

local net    = require "org.conman.net"
local clock  = require "org.conman.clock"
local signal = require "org.conman.signal"
local errno  = require "org.conman.errno"

local raddr = net.address("192.168.90.118",'udp',22222)
local sock  = net.socket(raddr.family,'udp')

signal.default('int')

signal.catch('alarm',function()
  sock:send(raddr,tostring(clock.get()))
end)

clock.itimer(1)

local previous = clock.get()

while true do
  local _,data,err = sock:recv()
  local now = clock.get()
  if data then
    local zen = tonumber(data)
    print(string.format("%.7f\t%.7f",now - zen,now - previous))
    previous = now
  else
    print(">>>",errno[err])
  end
end

and when we run it:

>>>	Interrupted system call
0.0003049	1.0015509
>>>	Interrupted system call
0.0002320	0.9998269
>>>	Interrupted system call
0.0002131	0.9999812
>>>	Interrupted system call
0.0001860	0.9999728
>>>	Interrupted system call
0.0002639	0.9999781

With POSIX, you can specify that for a given signal, system calls are to be automatically restarted so you can dispense with EINTR error handling.

And here's were we finally get to the “Oh, yeah, I didn't think that through, did I?” type of bug.

Not wanting the code to be interrupted by the alarm signal, I changed the call to signal.catch() so it would restart any system calls:

signal.catch('alarm',function()
  sock:send(raddr,tostring(clock.get()))
end,'restart')

When I ran the code, I got nothing! There was simply no ouput happening. It caught me by surprise and it took me several minutes to figure out what was happening (or rather, what wasn't happening):

We enter sock:recv() (a Lua function), which ultimately does a system call to recvfrom();
the alarm signal (SIGALRM) is raised;
the kernel iterrupts our current system call (recvfrom()) and calls the signal handler we installed;
the signal handler calls lua_sethook() so that the Lua VM (when it resumes) will be stopped and our Lua hook function (written in Lua) will execute;
the kernel then resumes the system call (recvfrom()) that was previously interrupted.

And thus we get to the punchline: the Lua VM doesn't resume because we're still in a system call! And thus, the signal handler written in Lua is never called, which doesn't send a packet, because we're stuck in our system call (recvfrom()) waiting for some data that will never arrive.

D'oh!

If the above code were written in C, there would be no issue; clock_gettime() and sendto() (the system calls underlying the Lua functions clock.get() and sock:send() respectively) are safe to call from a signal handler. I may not have been able to safely convert the time to text (since snprintf()—the only standard C function able to convert numbers to text, isn't documented as being safe to call in a signal handler) but sending the raw binary values would be okay in that case.

But this isn't C, it's Lua. And what we have here is a type of leaky abstraction. That 20/20 hindsight is such a bastard.

Obligatory Picture

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

Obligatory AI Disclaimer