The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, April 01, 2015

I solemnly swear that on this April Fools' Day I will not post a fake engagement photo on MyFaceGoogleBookPlusSpace like some kind of asshole. Nor will I perpetuate a celebrity death hoax. Because that makes people sad. In fact, I will not play any practical jokes. And if I see someone planning a prank, I will say to them: “Hey, you're being a dick right now. Stop being a dick. Stop being a dick. Seriously, stop being a dick.”

Seriously, take the pledge. The rest of the world will thank you for it.

---

Serializing Lua data, part III

Continuing the series on Lua serialization, we resume with the Lua userdata type. At this point, all I'm doing is conjecture, since I haven't actually bothered with serializing userdata and this post is about why I didn't bother.

The Lua manual has this to say about userdata:

The type userdata is provided to allow arbitrary C data to be stored in Lua variables. A userdata value represents a block of raw memory. There are two kinds of userdata: full userdata, which is an object with a block of memory managed by Lua, and light userdata, which is simply a C pointer value. Userdata has no predefined operations in Lua, except assignment and identity test. By using metatables, the programmer can define operations for full userdata values (see §2.4). Userdata values cannot be created or modified in Lua, only through the C API. This guarantees the integrity of data owned by the host program.

Already we're in trouble with respect to serialization. The light userdata is just a pointer, no more, no less. There is no indication of just how much memory it points to, much less what it points to or what the memory even represents. That's not to say that a light userdata has no meaning, but such meaning is only applicable to any code written in C (or C++) and not to Lua.

The serialization code gets a light userdata and … meh? There is nothing we can safely serialize about a light userdata. One answer might be to serialize the pointer value, but a pointer value only has meaning within an instance of a program. Run the same program again, and the value of the light userdata might not be the same (nor should you assume it'll be the same). Give that pointer value to another program, and who knows what it points to in that other program.

We could try reading the memory the light userdata points to, but as I said, there is no indication of how much memory we can read, if any. In general, serialization of light userdata is out of the question (I'm not saying it can't be done, I'm just saying I'm not writing a general purpose serializer that will attempt it).

A full userdata, while it still has no meaning for Lua, does have a length associated with it, so it is conceivable we could serialize the actual memory associated with a full userdata but again, we run into trouble. For instance, let's try to serialize the only standard Lua userdata—the result of opening up a file.

The userdata associated with a file in Lua is the following structure:

typedef struct luaL_Stream {
  FILE *f;
  lua_CFunction closef;
} luaL_Stream;

The first field is a pointer to a FILE, an opaque type (even to C programmers) that represents an “open file,” and sending the actual FILE value across is meaningless; one, because the contents differ from system (say, Microsoft Windows) to system (Linux), and two, it supposes that the “file” being opened exists when this is deserialized and used.

Now, given that CBOR allows additional semantics to be attatched to data, we could include the contents of the “open file” as part of the serialization and semantically tag the data as a “file, please create and open this.” We could, but there are a few issues even with this—what if the “open file” isn't a file at all? What if it's the output from a command? Or a network connection? How you do serialize a connected TCP connection? But say that we do have, in fact, a file, one that happens to be two gigabytes in size? Would you really want to serialize a two gigabyte file?

So solve those issues, there's still the issue of how the file was opened. Was it opened “read only?” We need to send a copy. “Write only?” No need to send anything. But the issue here is—there is no standard way to determine if a file was opened “read-only” or for “write-only” or even “read/write but writes are appended.” There are non-standard ways of doing it, but it varies from system to system.

Now, about that second field. It's a pointer to a C function, and as I stated before, serializing a Lua function in C is not possible. I worked around it by tagging the name of the function, but there's no name for this function, at least, one known to Lua. And why point to a hidden function to close the file? Because for files, you need to call the C function fclose(), but for a “file” opened by popen() (a function to run another process and either write data to it, or read data from it) you need to call the C function pclose(). That's why the function pointer.

And that's for a well known, standard Lua userdata. There are other types of userdata, depending upon the Lua modules in use, such as LPeg (where the full userdata for that represents an entirely different virtual machine geared for parsing text) or even my own org.conman.pollset (which is usually used to multiplex I/O with network connections, but can be used for other devices such as serial ports, and whose implementation depends on the operating system, as if things weren't difficult enough).

This also depends upon the module handing a given full userdata already loaded when deserializing. I'm not saying this is all impossible, but I do think it's beyond the scope of a general library to handle. Perhaps having the user register some type of callback to handle userdata might be required. I already support callbacks for tagged CBOR data, so adding support for full userdata using callbacks shouldn't be that hard.

All that's left—threads.

Thursday, April 02, 2015

Thank god that's over with!

At least yesterday is now over and those suffering from aphrilophobia can climb out from under their beds and go about their business for the next 365 days (next year being a leap year).

---

Serializing Lua data, part IV

The last Lua data type to serialize—threads.

It's more difficult than serializing a function, but probably a bit easier than serializing a userdata. Possibly. I haven't actually tried it yet, as it can't be done through Lua.

The issue is that there's no API to manipulate the callstack, and without that, you're stuck with dipping into the internals of Lua via C, which explains why most existing serialization modules haven't bothered with threads. And the only one that has attempted it, only works for Lua 5.1.

And that's why I didn't bother with it, either.

---

It was an interesting experience to write a serialization library for Lua. It became apparent exactly why so many don't bother with functions, userdata or threads—they're not easy to universally support.

I also skipped out on supporting metatables, not because they're hard, but because I just didn't get around to it. I also think that CBOR makes for a good serialization format. The primitives are well chosen, it's a consistent format and the semantic tagging makes it easy to extend.

And like I said earlier, I had fun playing around with this stuff.

Friday, April 03, 2015

A little snippet of permutations

I'm working on a little Lua project, and I found myself with a surprisingly hard problem. The task, take the following string:

one/{two three four five}/alpha/beta/{gamma delta epsilon}.c

and generate all possible permutations of the string with the words in the braces appearing once. In essence, generate the following list:

one/two/alpha/beta/gamma.c
one/two/alpha/beta/delta.c
one/two/alpha/beta/epsilon.c
one/three/alpha/beta/gamma.c
one/three/alpha/beta/delta.c
one/three/alpha/beta/epsilon.c
one/four/alpha/beta/gamma.c
one/four/alpha/beta/delta.c
one/four/alpha/beta/epsilon.c
one/five/alpha/beta/gamma.c
one/five/alpha/beta/delta.c
one/five/alpha/beta/epsilon.c

Parsing the string into a usable format was trivial (code left to the reader—hint: LPeg). So, starting from the output of parsing:

{
  "one/",
  {
    "two",
    "three",
    "four",
    "five"
  },
  "/alpha/beta/",
  {
    "gamma",
    "delta",
    "epsilon",
  },
  ".c"
}

Then … what?

You have to iterate through the main table, and at the same time, iterate through any subtables that might exist (anywhere from zero on up). It took me a while to come up with the code. I knew that there was an elegant way to do this, and by God, I was going to find it.

Several hours later, and:

-- ***************************************************************
--
-- Copyright 2015 by Sean Conner.  All Rights Reserved.
-- 
-- This library is free software; you can redistribute it and/or modify it
-- under the terms of the GNU Lesser General Public License as published by
-- the Free Software Foundation; either version 3 of the License, or (at your
-- option) any later version.
-- 
-- This library is distributed in the hope that it will be useful, but
-- WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
-- or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public
-- License for more details.
-- 
-- You should have received a copy of the GNU Lesser General Public License
-- along with this library; if not, see <http://www.gnu.org/licenses/>.
--
-- Comments, questions and criticisms can be sent to: sean@conman.org
--
-- ********************************************************************

-- ***********************************************************************
-- Lua 5.3 renamed the unpack() function to table.unpack().  So check for
-- Lua 5.3 and handle accordingly.
-- ***********************************************************************

if _VERSION == "Lua 5.3" then
  unpack = table.unpack
end

-- ***********************************************************************
-- This will take an array of strings and tables, and return an iterator
-- that will return successive permutations of strings.  
--
-- The table is unpacked into arguments, and the add() function will slowly
-- walk down the argument list, calling itself recursively to generate all
-- possible strings from the list of arguments and yield them one at a time.
--
-- The expand() function will create the coroutine and return a function
-- usable by the for keyword.
-- ***********************************************************************

function expand(list)
  local function add(a,x,...)

    -- ------------------------------------------------------------
    -- if x is nil, then we've exhausted all the paramters and have
    -- accumulated a string we can yield.  So yield it up.
    -- ------------------------------------------------------------

    if not x then
      coroutine.yield(a)

    -- --------------------------------------------------------------
    -- if x is a string, call ourselves with the contatenation of our
    -- accumulator string with x, and the rest of the paramters
    -- --------------------------------------------------------------
    
    elseif type(x) == 'string' then
      add(a .. x , ...)

    -- ----------------------------------------------------------------------
    -- otherwise, we have a table.  So iterate through the table, calling
    -- ourselves each time with an updated accumulator value and the rest of
    -- the parameters.
    -- ----------------------------------------------------------------------

    elseif type(x) == 'table' then
      for i = 1 , #x do
        add(a .. x[i], ...)
      end
    end
  end
  
  return function(co)
    local okay,res = coroutine.resume(co)
    return res
  end,coroutine.create(function() add("",unpack(list)) end)
end

-- ***********************************************************************
-- The parsing of 
--
-- 	one/{two three four five}/alpha/beta/{gamma delta epsilon}.c
--
-- into a table is left to the reader for an exercise.  For right now, I'm
-- using a hardcoded table.
-- ***********************************************************************

test = 
{
  "one/",
  {
    "two",
    "three",
    "four",
    "five"
  },
  "/alpha/beta/",
  {
    "gamma",
    "delta",
    "epsilon",
  },
  ".c"
}

-- ***********************************************************************
-- And now, just expand the list, printing each result.
-- ***********************************************************************

for path in expand(test) do
  print(path)
end

It's a decent example of recursion (where the trick is find the right base case and the reductions that lead to said base case). There might be non-recursive solutions, but I shudder at the potential complexity of such solutions.

Saturday, April 04, 2015

Some thoughts on make

From time to time, I find make limiting, think there has to be a better way and I start playing around with the idea of building a new make. It's not that uncommon for programmers to think this (SCons, CMake, ant, rake and a bunch of other programs) but oddly enough, as I try them, I keep going back to make (specifically, GNU make; note that when I mention make, I am talking about GNU make).

I think the main reason I go back is that make is mainly a syntax-light declarative langauge with bits of shell scripting thrown in. The other build systems are generally based around another language I do not know, they tend towards being very imperative, and rarely can you parallelize building the software (and when you get the makefile right, a parallel run of make just flies).

So when I had that “let's remake make” itch this time, I thought that perhaps I would see if I could stay within the confines of the existing syntax of make as much as possible. I wasn't trying to actually program my own make, but rather, just play around with how I would like make to look and work.

The main problem with make is declaring the dependencies. For instance, I'm embedding a UUID library into a larger project, and because recursive make is considered harmful, I include in my makefile:

lib/libspcuuid.a : build/third_party/uuid/luauuid.o	\
		build/third_party/uuid/uuid_ns_dns.o	\
		build/third_party/uuid/uuid_ns_null.o	\
		build/third_party/uuid/uuid_ns_oid.o	\
		build/third_party/uuid/uuid_ns_url.o	\
		build/third_party/uuid/uuid_ns_x500.o	\
		build/third_party/uuid/uuidlib_cmp.o	\
		build/third_party/uuid/uuidlib_parse.o	\
		build/third_party/uuid/uuidlib_toa.o	\
		build/third_party/uuid/uuidlib_v1.o	\
		build/third_party/uuid/uuidlib_v2.o	\
		build/third_party/uuid/uuidlib_v3.o	\
		build/third_party/uuid/uuidlib_v4.o	\
		build/third_party/uuid/uuidlib_v5.o

build/third_party/uuid/luauuid.o	: third_party/uuid/src/luauuid.c
build/third_party/uuid/uuid_ns_dns.o	: third_party/uuid/src/uuid_ns_dns.c
build/third_party/uuid/uuid_ns_null.o	: third_party/uuid/src/uuid_ns_null.c
build/third_party/uuid/uuid_ns_oid.o	: third_party/uuid/src/uuid_ns_oid.c
build/third_party/uuid/uuid_ns_url.o	: third_party/uuid/src/uuid_ns_url.c
build/third_party/uuid/uuid_ns_x500.o	: third_party/uuid/src/uuid_ns_x500.c
build/third_party/uuid/uuidlib_cmp.o	: third_party/uuid/src/uuidlib_cmp.c
build/third_party/uuid/uuidlib_parse.o	: third_party/uuid/src/uuidlib_parse.c
build/third_party/uuid/uuidlib_toa.o	: third_party/uuid/src/uuidlib_toa.c
build/third_party/uuid/uuidlib_v1.o	: third_party/uuid/src/uuidlib_v1.c
build/third_party/uuid/uuidlib_v2.o	: third_party/uuid/src/uuidlib_v2.c
build/third_party/uuid/uuidlib_v3.o	: third_party/uuid/src/uuidlib_v3.c
build/third_party/uuid/uuidlib_v4.o	: third_party/uuid/src/uuidlib_v4.c
build/third_party/uuid/uuidlib_v5.o	: third_party/uuid/src/uuidlib_v5.c

Or, if I wanted to save myself some typing:

define OBJECT_template = 
  $(1) : $(2)
endef

UUIDSRC = $(wildcard third_party/uuid/src/*.c)

lib/libspcuuid.a : $(subst third_party/uuid/src,build/third_party/uuid,$(UUICSRC))

$(foreach target,$(UUIDSRC),$(eval $(call OBJECT_template(build/third_party/uuid/$(notdir $(target)),$(target)))))

and have no idea what is going on three months from now (if indeed, I got that right). Instead, I would like to type:

lib/libspcuuid.a : build/third_party/uuid/*.o
build/third_party/uuid/*.o : third_party/uuid/src/*.c

and be done. This should be possible. “lib/libspcuuid.a” doesn't exist, but it depends upon all the “.o” files in “build/third_party/uuid”, which don't exist, but we have a rule that says “for ‘.o” files in ‘build/third_party/uuid’, there should be a corresponding ‘.c” file in ‘third_party/uuid/src/’ that is is generated from.” Which also leads to another problem with make: if “build/third_party/uuid” doesn't exist, make should make it! Automatically! I mean, make has no problems with “uuidlib_v5.o” not existing—that's the reason we're using make in the first place, to make files! If a directory a file lives in doesn't exist, make should make that too. Right?

By the same token, if I wanted to include some Lua modules but not all of them, I would like to do:

SPCMODC = env errno fsys hash math net pollset process strcore sys syslog
SPCMODL = debug getopt string table unix

lib/libspcmod.a : build/third_party/lua-conmanorg/*.o

build/third_party/lua-conmanorg/*.o : third_party/lua-conmanorg/src/{ $(SPCMODC) }.c \
	third_party/lua-conmanorg/lua/{ $(SPCMODL) }.lua

(Ah! So that's where yesterday's code snippet came from!)

I'll spare you the expansion.

Another small bit of annoyance is writing implicit rules to build the files. I learned the hard way that in make, this:

%.o : %.c
	$(CC) $(CFLAGS) -c -o $@ $<

only works if the “.c” and “.o” files are in the same directory! So the above rules works fine for:

long/path/to/src/foo.o : long/path/to/src/foo.c
src/bar.o              : src/bar.c
snafu.o                : snafu.c

But not for:

build/foo.o    : long/path/to/src/foo.c
build/bar.o    : src/bar.c
build/snafnu.o : snafu.c

Nope, I also have to write:

build/%.o : long/path/to/src/*.c
	$(CC) $(CFLAGS) -c -o $@ $<

build/%.o : src/%.c
	$(CC) $(CFLAGS) -c -o $@ $<

build/%.o : %.c
	$(CC) $(CFLAGS) -c -o $@ $<

if I want to segregate the “.o” files from the “.c” files.

How hard is it to realize that if I have a “.c” file, then the command to make a “.o” file is the same, regardless of where the “.c” file comes from, or where the “.o” file is being generated. Hello?

And while we're on that subject, one implicit rule I use is the following:

%.o : %.lua
	$(LUAC) -o $(@D)/$(*F).out $<
	$(BIN2C) -9 -o $(@D)/$(*F).c -t $(*F) $(@D)/$(*F).out
	$(CC) $(CFLAGS) -c -o $@ $(@D)/$(*F).c
	$(RM) $(@D)/$(*F).out $(@D)/$(*F).c

That bit of jibberish first compiles the Lua code using luac, then that output is converted to a C file which is then compiled into a object file so I can link it into the final executable, them removes the temporary files it needed to do all that. The noise you see is the cruft necessary to specify temporary files that won't get overwritten when doing a parallel build. Much nicer would be something like:

%.o : %.lua
	$(LUAC) -o $$.1 $<
	$(BIN2C) -9 -o $$.2 -t $(*F) $$.1
	$(CC) $(CFLAGS) -c -o $@ $$.2
	$(RM) $$.1 $$.2

(even better—make automatically deletes the intermediate files unless I tell it not to)

One complaint about make (or rather, makefiles themselves) is that a change to the makefile does not cause a recompile, mainly because the makefile itself is never listed as a dependency anywhere. This is never what you want! If the makefile was listed as a depenency (either explicitly or implicitly), then adding a new file to be compiled in would cause everything to be recompiled, when it should be the new file, any files that call code in the new file, and a final relinking of the code is all that is needed. Also, if you change a variable, like $(LDFLAGS), then all that should happen is any rule that uses $(LDFLAGS) should be run, not a complete build.

In other words, you really have a dependency on a portion of the makefile, not the makefile as a whole. But solving this would require a possible rethink of how make works (perhaps a cached version of the makefile that mode checks against, and intelligently applies the changes as dependencies? You know, so if you change $(CC), any rule using $(CC) is automatically run).

I realize these some of my ideas are not that easy to implement (heck, I wasted a few hours yesterday writing code to properly build a list of targets and dependencies based on ideas presented here), but I think they may go a long way to making make less horrendous to use.

Sunday, April 05, 2015

No Reservations in Narnia this fine Ēostre day

Ah, today's holiday. A celebration of the reanimation of a Jewish carpenter using Nordic symbols, named after a Germanic deity and involving sweets derived from Central American food. Ain't modern life wonderful?

As for me, I'm spending the day reading about Anthony Bourdain's eating his way through Narnia.

Monday, April 06, 2015

Someone bought Casa New Jersey and you have to see what happens next

When I left Casa New Jersey, it needed quite a bit of loving care. But sure enough, someone came along and lavished quite a bit of loving care on the house. And seriously, what a transformation! (link via Spring on GoogleMyFacePlusSpaceBook).

I especially like the work done upstairs. Wow!

Tuesday, April 07, 2015

Just use /dev/urandom and be done with it

I noticed one of the developers at The Ft. Lauderdale Office of The Corporation using the time of day to seed a random number generator, which is borderline okay (depending on how the resulting random numbers will be used) there are better ways to generate a random seed, at least on a modern POSIX system—read data from /dev/urandom.

My fellow cow-orker B, with whom I was having this discussion, mentioned this borderline paranoid approach to reading /dev/urandom. But I think that if you have to call fstat() to make sure the file is actually /dev/urandom then you have more things to worry about (really—if a cracker can substitute /dev/urandom with known data, it's pretty much game over—B agreed with that statement, by the way). Besides, the author wasn't paranoid enough! Who's to say there isn't some extra code in there (say, via $LD_PRELOAD or ptrace() or maybe even through some ELF magic on the executable) that intercepts the read() function to return “random data” when reading from /dev/urandom? Hmmmm? (about the only thing you can do to counter that is nuke the site from orbit—it's the only way to be sure)

But in the mean time, just use /dev/urandom.

Wednesday, April 08, 2015

Privacy? At GoogleMyFacePlusSpaceBook? Surely you jest!

I was inspecting Facebook's network traffic today in Firefox Devtools, when I realized that any text I put into the status update box was sent to Facebook's servers, even if I did not click the post button. Ever curious, I Googled this behaviour and came across a study which reveals some very frightening information:

Facebook calls these unposted thoughts "self-censorship," and insights into how it collects these nonposts can be found in a recent paper written by two Facebookers.

Via Lobsters, Facebook sending 'nonposts' to its servers and storing unpublished thoughts

I wouldn't be surprised if all the sites in the MyFaceGoogleBookPlusSpaceosphere aren't doing the same thing. And the thing is, when I do a search on “experimenting on unwilling subjects” the results all seem to be Na zis Nazis Nazis for some reason. Just approach FaceGoogleMyPlusSpaceBook that everything posted, whether “private” or not, is going to be seen by everybody. Because it is.

Thursday, April 09, 2015

Harry Potter and the Methods of Rationality

Finally!

Harry Potter and the Methods of Rationality is finally finished! I'm so happy!

I started reading this a few years ago, but the rate of new chapters was never that fast and it's only been in the past month or so that Eliezer Yudkowsky finished writing it. I think the premise is wonderful—that Harry Potter was raised by scientists and skeptically approached magic at Hogwarts by applying rational thought and scientific experimentation. It takes a while to start, but once it does, I found I could not stop reading it (well, until I reached the current chapter and had to wait a several months for the next outburst of chapters).

Over the past two nights, I've finished reading it (staying up way past my bedtime), and I must say, I found He-Who-Shall-Not-Be-Named to be a quite facinating character (when he finally did show up—oh, is that a spoiler?).

I think it's worth the read, all 122 chapters of it. Don't worry, it's a complete story with Harry winning and He-Who-Shall-Not-Be-Named losing and not a seven book series, so don't let the length discourage you if you are a Harry Potter fan—at least now you can finish it in one reading binge if you so desire.

Friday, April 10, 2015

Notes about a conversation overhead in the kitchen at Chez Boca

“Aaaaaaaaaah!”

“What happened?”

“The 16 ounce bottle of vanilla extract fell on the floor!”

“Did the bottle break?”

“No, but there's vanillia extract all over the pantry.”

“Was it the used bottle?”

“It's used now.”

“Is there any left?”

“About a quarter of the bottle is left.”

“Sigh. That was the good stuff.”

“At the very least, it will smell incredible in here for a while.”

Saturday, April 11, 2015

Coyote's Flying Saucer Retrievals and Repairs

It's nice to know that Coyote's Flying Saucer Retrievals and Repairs exists so when my C-57D needs repair, I know where to go to have it checked out. The only bad thing is that it is in California and not in Roswell or Rachel where all the flying saucer activities seem to happen in this country. I guess I'll have to keep in mind that if I crash my C-57D, I need to crash in or near Ocotillo, California.

Sunday, April 12, 2015

For me, SPF is not that great an anti-spam measure

I was reminded of SPF the other day. It's an anti-spam measure, primarily to help identify email spoofing. I set up an SPF record on my domain years ago, but other than that, I haven't really done anything else with it. But being reminded of it, I thought it might be a good idea to see just how effective it could be. I'm already using a greylist daemon to cut down on spam but hey, the more spam that is caught, the better.

First step—just how effective is the greylist daemon? I have the logs from the greylist daemon for the past month (March 15^th to April 11^th). Some processing on the logs and I have my answer:

Unique emails processed by the greylist daemon

Emails accepted	5,028
Emails rejected	5,132
Total	10,160

Wow!

A bit over 50% of the emails received by my email server are spam. I'm not sure if I should be depressed that easily half the email addressed to me is spam, or happy that the greylist daemon is an easy way to avoid false positives. I suppose both are in order.

Now, I still get spam despite the greylist daemon, but all that means is that the sender is actually bothering to follow the SMTP protocol—not that high a bar. So, of the emails that do get through, how much would get flagged with SPF? Okay, time to check up on the SPF specification, and boy, is that a mess. The grammar to parse SPF records requires backtracking (lovely!—and lest you think a message from 2010 has any relevence to a 2014 standard, think again; the grammar didn't change all that much) and not entirely context free either (sigh—one letter in the macro-expansion has two meanings depending on where it appears).

Oh, and that grammar? It's actually covers three different grammars—one for parsing the SPF record itself, a second one to parse an email header, and the third a secondary text string via a secondary DNS query (the SPF record itself is obtained via a DNS query, by the way).

Okay, so munging the grammar to what I think is intended and leaving out what I don't need, I went through the log file and for each accepted email, did an SPF check according to the specification. Granted, the data I get now might not reflect the results made at the original time, but it should give me a baseline to go by.

For the test, I pulled out all emails accepted (5,028) and removed those I explicitly allowed (for example, accept anything from a given IP address, or from a given domain) or that did not have a sender address (allowed by the SMTP protocol to prevent mailing loops when generating mail bounce messages), leaving me with 4,343 emails. Then, for those, I looked up the SPF record for the given domain, and if it had one, applied its policy.

The 4,343 accepted emails came from 1,000 unique domains, of which only 433 had an SPF record. Okay, 43% of the domains have an SPF record. And of the domains that had an SPF record, only 629 emails accepted were checked. Or 12½% of all accepted incoming emails could be checked via SPF. Sigh.

But of those that were checked via SPF, how did we fare? Were a lot spam? Or were most acceptable forms of email pef SPF policy?

Results of applying SPF policy against incoming email

fail	43	IP address was not allowed to send this email
softfail	53	IP address should not be sending this email (used for testing)
neutral	90	IP address has no policy
pass	443	IP address is allowed to send this email

A 70% pass rate for SPF. Only 43, or almost 1% (or around two per day) could have been deleted as spam. Another 53 maybe, possibly, could have been deleted as spam. And 90 no idea one way or the other. Sigh.

You want to know what has a better rate of catching spam than SPF for my email? Any email addressed to my domain registration email not from the registrar. For me, I don't think it worth the effort to implement this.

Monday, April 13, 2015

One of the most impressive demoscene videos I've ever seen

This is simply amazing.

I'm impressed. I didn't think it was possible to get more than 16 colors from the CGA card and here is a demo showing over 1000 colors. On a stock IBM 5150 PC from 1981 (video via 8088 MPH: We Break All Your Emulators « Oldskooler Ramblings, from a discussion at Hacker News). And for those of you who are curious as to how this was done, there're indepth articles about the graphics programming (via Hacker News) and the sound programming (again, via Hacker News).

Tuesday, April 14, 2015

Honey on tap

The most successful fully funded crowdfunding campaign on Indiegogo is not for a new smartwatch, video game, or 3D printer. It is for a new way to harvest honey, a potential breakthrough in a practice that has not seen a significant technological advance since 1852, when the Rev. Lorenzo Langstroth patented America's first movable frame beehive.

The Flow hive has amassed $8.9 million from more than 25,000 backers in one month (the goal was just $70,000), a possible sign that the urban-hipster revival of beekeeping is still alive, even as the U.S. honeybee population continues to die off. (The USDA has sounded the alarm, estimating that a third of all honeybees have died since 2006. The main suspect is a class of neuro-active insecticides called Neonicotinoids.)

Via Instapundit, This 'Honey on Tap' Beehive Design Just Raised $9 Million on Indiegogo

I've heard of this, but this is the first time I've seen how the Flow™ Hive works, and it's pretty ingenious. I know T, my fellow cow-orker, might be interested in this as he keeps bees as a hobby.

Wednesday, April 15, 2015

The taxman cometh!

Telonephobics beware! For it is the Ides of April and the second inevitability of life is upon us. But fear not if you lack health insurance and cannot afford $95 or 1% of your income (whatever is higher) since it is very easy to avoid paying the penalty tax:

#4. Get beaten by your wife. Required documentation: None
Official text reads: "You recently experienced domestic violence. Required documentation: None."
Translation: Get attacked by your wife. You may then apply for the exemption by simply stating “I am exempt because my wife beat me.” No documentation required.

Via Captain Capitalism, Citizen Liberty: 17 Ways To Avoid The Obamacare Tax Penalty, Including Being Beaten By Your Wife

It doesn't surprise me that a 2,700 page bill that no Congresscritter read and over 33,000 pages of regulations (wow! Something longer than Atlas Shrugged) would have egregious loopholes in it.

Really?

NO DOCUMENTATION?

Sheesh.

Thursday, April 16, 2015

How do I know you're allowed to send me this?

I was pulled into an improptu design meeting at work. Originally it was to discuss the format of a new URI for our Android application to use, but it quickly shifted into an authentication issue on the Android platform.

I don't work on Android applications (I do the call-processing on the telephony network side, not the cellphone side), so my terminology might be a bit off but the gist of the issue is our application, named Awesome Application (name changed to protect me), comes preinstalled on Android phones. We've allowed another preinstalled application, Bodacious Bronies (name completely made up), not written by us, to, when a certain action is done by the user in Bodacious Bronies, launches our application. This is done by Awesome Application listening for an intent sent by Bodacious Bronies, and then doing it's thing.

But the issue the our developer, D, had with this is to prevent the Malevolent Malcontent application (for example) from spamming Awesome Application with repeated intents. The fear here is that Malevolent Malcontent could so annoy the user with our program always popping up that the user would then uninstall Awesome Application, or worse, bitch, complain and moan to the phone carrier to remove or disable our application post-haste.

Our application could check the uid or the package name of the intent sender and only do the thing it does if the sending application is allowed, but neither the uid or the package name is fixed; either one can change with an update, and if our application isn't updated with the new uid or package name, then our app does nothing since it doesn't know the intent is from an allowed application.

And other methods, like having Bodacious Bronies sign the intent (somehow) is still subject to attacks; in order to sign the intent, a private key needs to be stored with Bodacious Bronies, (and we were sure that the creators of Bodacious Bronies would not want to include a private key with the application) and what's to stop the creators of Malevolent Malcontent from nabbing that private key (the developer of Malevolent Malcontent could get an Android phone, jailbreak it, and extract the Bodacious Bronies private key) and forge intents?

Security is hard. So is authentication.

There is a way to keep intents from being broadcast to every application. D was trying to find a way to avoid this, as one of our customers wanted the broadcast method of intents (for some reason—again, I'm not an Android developer so I'm not sure what the trade-offs are here) but he decided that the best course of action is to use the non-broadcast intent method. Now he has to convince the Powers-That-Be that this is the only way.

Friday, April 17, 2015

Minnesota's great umlaut war is over

Today, we bring you urgent and breaking news out of Minnesota, where a battle over umlauts has been — well, not raging. What is the more polite version of raging? Occurring? Happening? Gently taking place? Something like that.

Anyway! Minnesota. Umlauts. See, there is a city in Minnesota that had been known as Lindström — or, if you saw the signs greeting you on the way in or out of town in recent years, Lindstrom.

Via Brian Yoder on MyFaceGoogleBookPlusSpace, Minnesota’s great umlaut war is over (also, Minnesota was having an umlaut war) - The Washington Post

My first thought was couldn't the MDOT just spell it “Lindstroem?” But then I read that Lindström has a sister city in Sweden, Tingsryd, and I wasn't sure if the umlaut served the same function in Swedish as it did in German. It turns out it doesn't, and the “ö” in Swedish is a distinct character, unlike in German where the “ö” is a shorthand notation for “oe.”

It all turned out fine though, the MDOT is going around adding umlauts on all the Lindström signs.

---

Ms. L'Engle also never mentioned the 24-hour news cycle that helped subjugate the population of Camazotz

Via a link on FaceGoogleMyBookPlusSpace is an article about a cut passage from an early draft of A Wrinkle In Time. The article talks briefly about the cut passage and then goes into some details about Madeleine L’Engle, but I can't help but quote from the cut passage:

So she said, “But Father, what's wrong with security? Everybody likes to be all cosy and safe.”

“Yes,” Mr. Murry said, grimly. “Security is a most seductive thing.”

“Well—but I want to be secure, Father. I hate feeling insecure.”

“But not enough you don't love security enough so that you guide your life by it, Meg. You weren't thinking of security when you came to resuce me with Mrs Who, Mrs Whatsit, and Mrs Which.”

“But that didn't have anything to do with me,” Meg protested. “I wasn't being brave or anything. They just took me.”

Calvin, walking beside them with his load of wood, said, smiling warmly at Meg, “Yes, but when we got here you didn't go around whining or asking to go home where you could be all safe and cosy. You kept yelling, where's Father, take me to Father: You never gave a thought to security.”

“Oh,” Meg said. “Oh.” She brooded for another moment. “But I still don't see why security isn't a good thing. Why, Father?”

“I've come to the conclusion,” Mr. Murry said slowly, “that it's the greatest evil there is. Suppose your great great grandmother, and all those like her, had worried about security? They'd never have gone across the country land in flimsy covered wagons. Our country has been greatest when it has been most insecure. This longin sick longing for security is a dangerous thing, Meg, as insidious as the strontium 90 from our nuclear explosions that worried you so about Charles Wallace when you read in science at school that it was being found in greater and greater quantities in milk. You can't see strontium 90. You can't feel it or touch it. But it's there. So is the panicky searching for conformity, for security. Maybe it's because of the Black Thing, Meg. Maybe this lust for security is like a disease germ that it has let loose on our land. I don't know, Meg. All I realize now is that my fight is much bigger than this little one on Camazotz.”

Despite being written over fifty years ago, it seems to apply more to us today than it did in 1962 (and here's a discription of Camazotz if you are unfamiliar with the book).

Saturday, April 18, 2015

Sadly, yet accurately, still no flying cars

Tell me, does any of this sound familiar?

(1) Random acts of violence by crazy individuals, often taking place at schools …

(2) The other major source of instability and violence comes from terrorists, who are now a major threat to U.S. interests, and even manage to attack buildings within the United States.

(3) Prices have increased sixfold between 1960 and 2010 because of inflation. …

(4) The most powerful U.S. rival is no longer the Soviet Union, but China. However, much of the competition between the U.S. and Asia is played out in economics, trade, and technology instead of overt warfare.

(5) Europeans have formed a union of nations to improve their economic prospects and influence on world affairs. In international issues, Britain tends to side with the U.S., but other countries in Europe are often critical of U.S. initiatives.

(6) Africa still trails far behind the rest of the world in economic development, and Israel remains the epicenter of tensions in the Middle East.

(7) Although some people still get married, many in the younger generation now prefer short-term hookups without long-term commitment.

(8) Gay and bisexual lifestyles have gone mainstream, and pharmaceuticals to improve sexual performance are widely used (and even advertised in the media).

(9) Many decades of affirmative action have brought blacks into positions of power, but racial tensions still simmer throughout society.

(10) Motor vehicles increasingly run on electric fuel cells. …

(11) Yet Detroit has not prospered, and is almost a ghost town because of all the shuttered factories. However. a new kind of music … has sprung up in the city.

(12) TV news channels have now gone global via satellite.

(13) TiVo-type systems allow people to view TV programs according to their own schedule.

(14) Inflight entertainment systems on planes now include video programs and news accessible on individual screens at each seat.

(15) People rely on avatars to represent themselves on video screens …

(16) Computer documents are generated with laser printers.

(17) A social and political backlash has marginalized tobacco, but marijuana has been decriminalized.

Oh, and let's not forget President Obomi.

Wait—what?

What you read was eighteen predictions (link via Hacker News) made by John Brunner is his 1969 novel Stand on Zanzibar. It's an incredible list, scarily accurate in its portrayal of life in 2010. I never read that book, but I did read Shockwave Rider that predicted a global network besieged with malware and The Sheep Look Up, a book about global environmental collapse that was the single most scary book I've ever read (that I try not to think about too much least I start having nightmares again). Both of those were very good (even if The Sheep Look Up is too horrifying to think about), so I would think Stand on Zanzibar would be great as well.

Sunday, April 19, 2015

The man behind the bits

This self-portrait:

also happens to be a valid QR code. Go head, try it if you can. See how deep the rabbit hole goes.

For the curious, I used QArt Coder to generate the image. And for the really curious, the theory behind how it works.

Monday, April 20, 2015

Just because it was that type of day

Because a picture of a bunny with a pancake on its head is cliché.

Tuesday, April 21, 2015

Notes on an overheard conversation in The Ft. Lauderdale Office of The Corporation

“Do you know how I get to the timesheet application?”

“Timesheet application? You want to apply for some vacation time?”

“Yes.”

“Okay, first you need to log into the VPN.”

“But we're in the office, why do I need to log into the VPN?”

“No, you're thinkging of The Corpration VPN. You need to log into the Corporation Overlord Corporation VPN.”

“Oh. Where do I go to log into that?”

“Here, let me email you the location.”

“Thank you. Hmm … Ah, I use that account name … but I don't seem to know the password for that account. Is it the same as any other password you use?”

“No, it's a different password.”

“Sigh. … Okay it's not that password … and it's not that password. I'm afraid of trying it a third time lest I get locked out.”

“So once you reset your password and can log on, do not select the ‘Timesheet Application’ but instead select the ‘Monopolistic Database Corporation Application Suite’ instead.”

“Do tell.”

“Yeah, I can't make this stuff up even if I wanted to.”

“So skip ‘Timesheet Application’ and instead use the ‘Monopolistic Database Corporation Application Suite.’”

“Yes.”

“I won't ask.”

“Good, beacuse I don't know the answer.”

---

Notes on another overheard phonecall in The Ft. Lauderdale Office of The Corporation

“Okay, I copied down the new password.”

…

“Wait? I'm stuck with that password?”

…

“Oh, I can't change the password for ten days. Then I can change it to something less obvious.”

…

“Okay, fine. Now let me try logging into the site.”

…

“No, I don't have Internet Explorer.”

…

“No, I'm not running Windows 7, or any version of Windows for that matter.”

…

“A Macintosh.”

…

“No, really.”

…

“No, I'm not running Google Chrome.”

…

“Firefox.”

…

“Is there a joke I didn't get?”

Wednesday, April 22, 2015

Surely you can't be serious

A couple of months ago, I was at a party somewhere, and a boy came up to me who was, like, 8 or 10 years old, and he said, “Oh, I really liked Airplane! I thought it was really funny!” And I said, “How was it that you came to see it?” And he said, “Well, my grandfather made me watch it.” [Laughs.] If you’d told us in 1980 that the grandkids of the audience would be the ones who’d keep the movie going, it would’ve been very gratifying. But I don’t think we ever anticipated it. And it’s one of the great thrills, I think, of all of our lives that it still remains well known.

Via Instapundit, Surely you can’t be serious: An oral history of Airplane! · Oral History · The A.V. Club

It's an oral history of the making of “Airplane!” from many people who were involved in the making of one of the funniest movies to ever come out of Hollywood. And stop calling me Shirley.

Thursday, April 23, 2015

Gotta love spam with a legal disclaimer

From

"John" <yjonjens@mail.com>

To

sean@conman.org

Subject

business leads

Date

Thu, 23 Apr 2015 17:22:50 +0200

Hey,

You are receiving this email because we wish you to use our email marketing service.

We wish to be your email marketing partner, we can grow your business 2-5 times than now.

If you would require more information please send us an email and we would be glad to discuss the project requirements with you soon. Looking forward to your positive response.

Kind Regards
John
Email: pottleyo@aliyun.com

---

This e-mail message and its attachments (if any) are intended solely for the use of the addressee(s) hereof. In addition, this message and the attachments (if any) may contain information that is confidential, privileged and exempt from disclosure under applicable law. If you are not the intended recipient of this message, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. Delivery of this message to any person other than the intended recipient is not intended to waive any right or privilege. If you have received this message in error, please promptly notify the sender and immediately delete this message from your system.

If you don't wish our future news letter, pls send address to ttickmay@aliyun.com for removal.

I'm only reproducing this because of the disclaimer. Really? You're fishing for clients, not giving legal, medical or confidential information. It doesn't mean a thing. Also, when I tried searching for this, Google helpfully mentioned:

Did you mean: This e-mail message and its attachments (if any) are intended solely for the use of the addressee(s) thereof. In addition, this message and the attachments (if any) may contain information that is confidential, privileged and exempt from disclosure under ap…

Heh.

Also, that text shows up on a lot of emails. A lot.

I would like to note that this came from 135328.com, a domain registered in China, from a server in Williamsville, NY. The email was from yjonjens@mail.com, which is from a domain registered to a company in Chesterbrook, PA and administered by a German company in Karlsruhe. The default Reply-To: address is broling@aliyun.com, which is from a copmany in China registered by what appears to be either a European or American. And as you can see, it doesn't match the “sender” address, nor the address mentioned in the email itself. I'm not worried about being sued by these jokers.

Friday, April 24, 2015

Notes about a Lua lint checker

Luacheck is a static analyzer and a linter for Lua. Luacheck detects various issues such as usage of undefined global variables, unused variables and values, accessing uninitialized variables, unreachable code and more.

Luacheck

The one real issue I have with Lua is its dynamic typing. Of all the bugs I fix in my own Lua code, I would say that the majority are due to typos (wrong variable name) or an unexpected type. So I was quite happy to come across and try out Luacheck. And fortunately, it's pretty straightforward to run.

I ran it over “Project: Sippy- Cup” and … wow. The extensive regression test I have has already flushed out the typos and the unexpected type errors I tend to make. But Luacheck found quite a few unused variables (which is nice—it also found a bunch of unsused LPeg expressions) and a ton of unintentional global variables (because I forgot to declare them with local).

The output is easy to read (here's a representative sample from some non- work related code I have):

Checking ptest-cr-select.lua                      Failure

    ptest-cr-select.lua:53:9: variable amount was previously defined as an argument on line 52
    ptest-cr-select.lua:128:9: variable okay is never accessed
    ptest-cr-select.lua:193:40: unused argument event
    ptest-cr-select.lua:197:43: shadowing upvalue conn on line 194
    ptest-cr-select.lua:213:21: shadowing upvalue argument event on line 193
    ptest-cr-select.lua:215:15: unused variable rem
    ptest-cr-select.lua:215:15: shadowing upvalue rem on line 194

Total: 7 warnings / 0 errors in 1 file

About the only false positive it finds is this idiom:

function foo(param1,param2)
  local param1 = param1 or "default value"
  local param2 = param2 or 3
  local a = ...
  -- ...
end

where it will flag param1 and param2 as shadowing an upvalue. This idiom though, is used to provide a default value if a parameter isn't given to a function. It's easy enough to fix, either:

function foo(param1,param2)
  param1 = param1 or "default value"
  param2 = param2 or 3
  local a = ...
  -- ...
end

or

function foo(param1,param2)
  local param1 = param1 or "default value" -- luacheck: ignore
  local param2 = param2 or 3 -- luacheck: ignore
  local a = ...
  -- ...
end

Overall, I'm glad I found this tool. It's been a real eye opener.

Saturday, April 25, 2015

Jamie and Adam Unleased.

Bunny and I just saw Jamie & Adam UNLEASHED at the Kravis Center. What a fun show. Adam and Jamie would select crash test dummies victims volunteers (and there were no shortage of thoses) to come up on stage to help demonstrate some principle of physics, such as a nine year old girl lifting two grown men a foot above the stage, or arranging four men in a sitting position without chairs, and other physics-based tricks.

They also talked about several myths they've done on their show Mythbusters which included lots of explosions. A lot of explosions. Including a several minute clip of various things they've exploded over the years (like water heaters, cars, buildings, cement trunks, more cars and in general, nearly every type of explosive device you can conceive of) that nearly brought down the house (literally, since they boosted the base so you could feel the explosions rattling the theater).

And to end the show, they shot a volunteer with a paintball gatling gun for what seemed like a solid minute (don't worry—the volunteer was wearing the suit of armor Adam wore to protect him underwater from sharks) leaving one paint covered volunteer and a volunteer shaped space on the wall behind him.

Very amusing stuff.

But I think the most amusing thing to happen at the show happened during intermission. I received the following text message:

I didn't recognize the number, and it took me a few moments to decide to even answer “Yes.” The response to my response was classic, and indeed, when I looked to my right, I saw my old roommate Rob, his wife Laura, Squeaky and his wife Tanya, sitting at the other end of the aisle.

'Tis a small world indeed.

Sunday, April 26, 2015

Just a quick note on something I noticed going on today on the home network

The script kiddies are active tonight.

Chain ssh-block (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   17  1812 REJECT     all  --  *      *       188.135.202.39       0.0.0.0/0           reject-with icmp-port-unreachable 
   38  2272 REJECT     all  --  *      *       113.106.85.23        0.0.0.0/0           reject-with icmp-port-unreachable 
    4   348 REJECT     all  --  *      *       117.253.105.235      0.0.0.0/0           reject-with icmp-port-unreachable 
   19  2080 REJECT     all  --  *      *       37.190.87.219        0.0.0.0/0           reject-with icmp-port-unreachable 
   20  2316 REJECT     all  --  *      *       187.72.49.52         0.0.0.0/0           reject-with icmp-port-unreachable 
   16  1796 REJECT     all  --  *      *       201.75.109.180       0.0.0.0/0           reject-with icmp-port-unreachable 
  512 25388 REJECT     all  --  *      *       218.83.6.81          0.0.0.0/0           reject-with icmp-port-unreachable 
   20  2248 REJECT     all  --  *      *       177.70.122.255       0.0.0.0/0           reject-with icmp-port-unreachable 
   15  1800 REJECT     all  --  *      *       117.253.215.122      0.0.0.0/0           reject-with icmp-port-unreachable 
   17  2032 REJECT     all  --  *      *       117.244.25.226       0.0.0.0/0           reject-with icmp-port-unreachable 
   18  2048 REJECT     all  --  *      *       134.255.165.240      0.0.0.0/0           reject-with icmp-port-unreachable 
   17  1964 REJECT     all  --  *      *       187.49.248.42        0.0.0.0/0           reject-with icmp-port-unreachable 

These are just the script kiddies caught trying to brute force a login to my home machine over the past hour (they're blocked after five attempts, and the block remains for an hour lest I end up with hundreds of entries). I wonder if there's a quota they have to meet?

Monday, April 27, 2015

A shot to the arm

I sit down. As soon as I do, the phlebotomist drops a padded bar across the seat. Sure, they claim its for resting your arm while they draw blood, but it's real purpose is to keep people from escaping. Most people don't hear the soft click as it's locked into position.

Then my arm is wrapped with a large rubber band, cutting circulation to my hand. Again, the claimed reason is to help a vein rise to the surface of the arm, but in reality, it's there to weaken the arm so you can't fight.

I'm surprised that I'm letting them do this to me. There must be something in the air to keep me docile throughout the procedure.

“This won't hurt a bit.” A lie. Not quite as bad as “the check is in the mail,” or “I'm from the government, I'm here to help,” but it's still a lie.

“Aaaaaaaaaaaaaaaaaiiiiiiiiiiiiiiiiieeeeeeeeeeaaaaaarrrrrrrrrrrrrrg!”

“I haven't started yet.”

“Just pract—aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaah!”

“Just a few moments more.”

“Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhhhh!”

“Be patient.”

“Errrrrrrrrrrrrr—are you a phlebotomist, or a vampire?”

“That remark cost you three more vials.”

“Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaarrrrrrrrrrrreeeeeeiiiiiiii!”

“There we go, that wasn't so bad, was it?”

“Aaaaaaaaaaaaaaaaaaaaaaaaaah!”

“The needle is out.”

“Aaaaaaaaaaaa—oh.”

“Here, hold this cotton pad here.”

I reach over with my other, non-blood-starved hand and place it over the cotton pad to staunch the flow of blood. The phlebotomist then tapes the cotton pad and my hand to my arm. “Um, you taped my hand to my arm.”

A very subtle click as the bar across the chair is lifted. “You are free to go.”

“Um. My hand is still taped to my arm.”

“Just pull it off. It won't hurt.”

Tuesday, April 28, 2015

Breaking Brain

Everybody knows how to ride a bike, right? But not everybody unlearns how to ride a bike. But Destin Sandlin did just that (link via Jason Kottke and no, it did not involve injury or a rare disease—he did this intentionally). He also untaught his six-year old son, who proved more adaptable because he was younger.

So, how do you unlearn riding a bike? Well, first off, you get a bicycle that steers backwards …

Wednesday, April 29, 2015

Alien packets from outer space

To help test “Project: Sippy-Cup,” I wrote a mock component to return precanned data releated to our test data. The mock component is basically a specialized DNS server that only expects a certain type of query. I wrote it so it would be easier to configure and run than a full blown bind installation.

But for the past few weeks it was crashing and pretty much the only report I would get back is “it crashed.” Sigh. I did find the error path in the mock component to be a bit spotty, but this was never intended to be a full blown product but rather something that would work just well enough to get the testing done. I never tested it with bogus queries because I never expected it to get bogus queries. We had control over all the data. The mock component would only talk to programs we were testing.

Even after fixing some “how did I not see that error?” type problems, the mock component was still crashing, and the only way that could happen is if the queries being sent were too large (over 512 bytes), the query was corrupted or malformed and could not be decoded, or if the query wasn't the single query type supported by the mock component. And there was no way any of that could happen. We controlled all the data!

Or so I thought.

It turns out the mock component was receiving God knows what from random computers on the Internet, which is incredible when the computer the mock component is running on doesn't have a public IP address!

Well, okay, it does have a public address, but it's a public IPv6 address, but the queries causing the crashes were all coming from IPv4 addresses.

Wow.

About the only thing I got to explain that behavior is the IPv6 address is routed via a tunnel, and perhaps there's some routing leakage that lets public IPv4 packets through. Other than that, I got nothing.

At the very least, I did fix the dodgy error handling so the mock component doesn't crash from data that it shouldn't get.

---

A trip to the hospital

Bunny was in the process of cutting up an old computer desk on the back porch with a sawzall when a chunk fell off in such a way not to break her big toe, but rip the nail off of it (ugh).

She managed to bandage it up and off we headed to the non-emergency emergency room at the local hospital. Bunny said it was more of a clinic than an emergency room but that it happens to be located in the emergency room.

Ooookay.

I drop her off, then spent the next ten minutes trying to locate the parking lot. Turns out that at the Boca Raton Regional Hospital, the emergency room parking lot (as well as the non-emergency emergency room parking lot) was across the street, marked by a few tiny signs with a low albedo.

Oddly enough, the emergency room wasn't crowded, and thus non-emergency emerency room wasn't crowded. By the time I had arrived in person, Bunny was already in the bowels of the hospital being treated.

And I'm stuck in the waiting room watching “Jurassic Park III.”

---

Limping, but at home

A bunch of people got eaten. Sam Neil managed to escape being eaten. The dinosaurs still roam that Central American Island like they own the place or something. And Bunny finally limped out of the non-emergency emergency room.

One clunky foot brace, some good medicine, dinner from Denny's (slogan: “You never start out for us, but you'll always end up here”) and all is, mostly, right with the world.

Thursday, April 30, 2015

Some perils of handing signals in Lua

This is one of those “Oh, yeah, I didn't think that through, did I?” type of bug.

I wrote a signal module for Lua, which can handle both ANSI C and POSIX signals with largly the same API (the POSIX implementation one has some additional functions defined).

Handling signals in Lua is not that straightforward because of the nature of signals—you are effectively writting multithreaded code. You just can't call back into Lua from the signal handler (while the Lua VM has no static data and each Lua state is isolated unto itself, two threads sharing a Lua state can lead to problemss). The only Lua function you can safely call is lua_sethook(), which can be used to stop the Lua VM at the next VM instruction (it's typically used for debugging and signal handing). This callback can then call back into Lua. It is a bit convoluted (the signal handler will call lua_sethook() and return; the Lua VM will resume and then call the hook), but it does allow you to write signal handlers in Lua:

signal.catch('windowchange',function()
  print("Wheeee!  Our terminal just resized!")
end)

and not have it blow up on you.

So, with that in mind, I give you this code:

local net    = require "org.conman.net"
local clock  = require "org.conman.clock"
local signal = require "org.conman.signal"

local raddr = net.address("127.0.0.1",udp,'echo')
local sock  = net.socket(raddr.family,'udp')

signal.catch('alarm',function()
  sock:send(raddr,tostring(clock.get()))
end)

clock.itimer(1)

local previous = clock.get()

while true do
  local _,data = sock:recv()
  local now = clock.get()
  if data then
    local zen = tonumber(data)
    print(string.format("%.7f\t%.7f",now - zen,now - previous))
    previous = now
  end
end

This is a UDP echo client program. signal.catch() handles the alarm signal (SIGALRM) by sending a packet of data (which is just the current time) to the echo server. clock.itimer() informs the kernel to send the alarm signal once a second. So once a second, our program receives the alarm signal and sends the current time. Then, in an infinite loop, we just wait for packets to arrive (which should be the packets we sent to the echo server—they're “echoed” back to us) and we calculate how long the packet took round trip and how long it was from the previous packet. The output looks like:

0.0002971	1.0014961
0.0003922	0.9999950
0.0002851	0.9998930
0.0003171	1.0000319
0.0003910	0.9999740
0.0002551	0.9998641
0.0003359	1.0000808

The first column is the round trip time (in seconds) for the packet (around 3 to 4 ten thousandths of a second), and the second column is how long (in seconds) from the previous packet (a second, give or take a few ten thousandths).

But our call to sock:recv() is interrupted by the alarm signal. Unfortunately, one side effect of signals is that they will interrupt “long running” system calls, which is almost always system calls dealing with I/O, such as read() or write(). When such a call is interrupted, the system call will return an error of EINTR. We can see this if we change the code a bit:

local net    = require "org.conman.net"
local clock  = require "org.conman.clock"
local signal = require "org.conman.signal"
local errno  = require "org.conman.errno"

local raddr = net.address("192.168.90.118",'udp',22222)
local sock  = net.socket(raddr.family,'udp')

signal.default('int')

signal.catch('alarm',function()
  sock:send(raddr,tostring(clock.get()))
end)

clock.itimer(1)

local previous = clock.get()

while true do
  local _,data,err = sock:recv()
  local now = clock.get()
  if data then
    local zen = tonumber(data)
    print(string.format("%.7f\t%.7f",now - zen,now - previous))
    previous = now
  else
    print(">>>",errno[err])
  end
end

and when we run it:

>>>	Interrupted system call
0.0003049	1.0015509
>>>	Interrupted system call
0.0002320	0.9998269
>>>	Interrupted system call
0.0002131	0.9999812
>>>	Interrupted system call
0.0001860	0.9999728
>>>	Interrupted system call
0.0002639	0.9999781

With POSIX, you can specify that for a given signal, system calls are to be automatically restarted so you can dispense with EINTR error handling.

And here's were we finally get to the “Oh, yeah, I didn't think that through, did I?” type of bug.

Not wanting the code to be interrupted by the alarm signal, I changed the call to signal.catch() so it would restart any system calls:

signal.catch('alarm',function()
  sock:send(raddr,tostring(clock.get()))
end,'restart')

When I ran the code, I got nothing! There was simply no ouput happening. It caught me by surprise and it took me several minutes to figure out what was happening (or rather, what wasn't happening):

1. We enter sock:recv() (a Lua function), which ultimately does a system call to recvfrom();
2. the alarm signal (SIGALRM) is raised;
3. the kernel iterrupts our current system call (recvfrom()) and calls the signal handler we installed;
4. the signal handler calls lua_sethook() so that the Lua VM (when it resumes) will be stopped and our Lua hook function (written in Lua) will execute;
5. the kernel then resumes the system call (recvfrom()) that was previously interrupted.

And thus we get to the punchline: the Lua VM doesn't resume because we're still in a system call! And thus, the signal handler written in Lua is never called, which doesn't send a packet, because we're stuck in our system call (recvfrom()) waiting for some data that will never arrive.

D'oh!

If the above code were written in C, there would be no issue; clock_gettime() and sendto() (the system calls underlying the Lua functions clock.get() and sock:send() respectively) are safe to call from a signal handler. I may not have been able to safely convert the time to text (since snprintf()—the only standard C function able to convert numbers to text, isn't documented as being safe to call in a signal handler) but sending the raw binary values would be okay in that case.

But this isn't C, it's Lua. And what we have here is a type of leaky abstraction. That 20/20 hindsight is such a bastard.

Friday, May 01, 2015

Living with being dead

Most of the patients here have been diagnosed with garden-variety neurological disorders: schizophrenia, dementia, psychosis, severe depression, or bipolarism. But the ones I am searching for are different. They suffer from an affliction even more puzzling: They believe that they are dead.

It’s a rare disorder called Cotard’s syndrome, which few understand. For patients who have it, their hearts beat and lungs pump, yet they deny their existence or functionality of their bodies, organs or brains. They think their self is detached.

Via Hacker News, Living With Being Dead — Matter — Medium

I would like to congratulate Sean Hoade on his Zombi epalooza interview, and what better way to do that than to link to an article about people who think they're dead. Does that mean they think they're zombies? Or ghosts? Or just dead and their body has yet to notice?

Saturday, May 02, 2015

Borg

Sigh.

I hate web based applications, because as soon as you get used to the interface—BAM some attention-deficit programmers change how everthing works, just because. Google Maps is a good example of this. It's still perhaps the best mapping application out there and I always use it, but every few months they change how the entire interface works, destroying existing patterns of use and wasting days, nay weeks of time as I attempt to learn how to use the features I use, only to find out half of them have been removed, because.

Ahhhhhhhhh!

But today I'm not here to bury Google Maps, but Facebook. They broke my posting application. The application I use when I post to this blog and send notification to Facebook that is posted on my … whatever that thing is called at Facebook. My wall? Timestream? Spam channel? Whatever it's called.

Facebook changed how things work on the backend, and now I'm getting the dreaded 803 error (and of course there are no real answers there).

Thank you Facebook.

Thank you a lot.

---

Amoeba

It appears that Facebook wants to be the internet (much like Google in fact) or at the very least, force the impression that the web is Facebook. Why else make such a drastic change in API that disallows small blogging sites from updating Facebook remotely?

After spending several hours pouring over the Facebook API documentation, my eyes are glazing over and from what I can see, it appears Facebook only supports three use cases (aisde from using the Facebook website itself):

1. an application running on Android;
2. an application running on iOS;
3. an application running on a website (preferrably using Facebook to authenticate the user).

And that last one—it's someone actively using the website. My now-broken application? That was kicked off when I posted to my blog (most of the time that's via email, where I can use an editor of my choice to compose the entry instead of whatever hideous crap editing you get in a TEXTAREA on a webpage) where I may or may not be logged into Facebook at the time (usually not, not that it matters at all for tracking purposes, which is for another post).

And my application wasn't the only one Facebook broke. And that appliation looks like it won't be fixed any time soon ever! (sorry Dan)

So it looks like I'm stuck manually posting to Facebook when I update here. I was already updating GooglePlus manually because they have yet to provide an API to update remotely (I don't expect one any time soon). I suppose I could automatically update Twitter, which can update Facebook (now that I worked around the broken Twitter API—are you seeing a pattern here?) but there's no telling how long that will last; I'd be stuck with a 140 character limit including the link and well … no.

There's more to the web than just GoogleMyFacePlusSpaceBook, and long term, I think it'll be easier to just manually update FaceGoogleMyBookPlusSpace. XXXX the GoogleMyFaceSpaceBookTwitterPlus APIs. XXXX them all!

Sunday, May 03, 2015

When Las Vegas is not Las Vegas

Ah, Lost Wages. It's been a few years since my last visit, but each time I've been there, I technically wasn't in Las Vegas, but in Paradise. Paradise, Nevada to be precise.

And yes, it was a tax dodge.

Monday, May 04, 2015

Today my friends are talking with a lithp.

So … does this mean The Empire should have installed a cup on the Death Star?

May the Fourth be with you!

Tuesday, May 05, 2015

Ch-ch-ch-changes for the mobile web

It's been brought to my attention by a few parties that my blog was unviewable on some smartphones; which smartphones I don't know (but I suspect Android based devices). I finally got around to it and the changes were minimal. This:

<meta name="HandHeldFriendly" content="True">

(the Google Mobile-Friendly Test fell on the floor laughing when it encountered that line) changed to:

<meta name="viewport" content="width=device-width, initial-scale=1">

And that's it for the HTML changes. The CSS changes weren't that bad, once I figured out what was needed. I asked a fellow cow-orker, D, what I needed to do in order to serve up a “mobile-friendly CSS file” and his advice was: “Do whatever CNN does.”

Sigh.

It appears there is no real reliable way to detect a smartphone through CSS only. Sure, I could try to detect a smartphone by sniffing the user agent, but I wanted something easy, not something error prone despite a ton of ongoing configuration and testing. So that was out. And the obvious media query:

<link media="handheld" rel="stylesheet" href="/CSS/smartphone.css" type="text/css">

was right out because “smartphones” are “smart” and not at all a “handheld.” Sheesh.

I ended up doing what CNN did—base the style upon the width of the browser. It seems that a “safe” width for smartphones is around 736 pixels. Larger than that, and I can assume a real desktop (or laptop) display; that or less and hey, treat it as a smartphone. And if your browser window on the desktop (or laptop) is less than 737 pixels, you'll get the “mobile” version of my site.

Anyway, the changes weren't all that bad. The “not-main-content” is positioned via CSS and that's all I really wanted to change. For instance, I had this style for the main content:

/* Yes, the DIV is redundant.  I left it in because I want to be explicit */
DIV#content
{
  margin-top:           0;
  margin-bottom:        0;
  margin-left:          220px;
  margin-right:         180px;
  border:               0;
  padding:              0;
}

To fix this for the smartphone:

DIV#content
{
  margin-top:           0;
  margin-bottom:        0;
  margin-left:          220px;
  margin-right:         180px;
  border:               0;
  padding:              0;
}

/* override some previous settings for "smartphones" */
@media screen and (max-device-width: 736px),
        screen and (max-width: 736px)
{
  DIV#content
  {
    margin-left:        0;
    margin-right:       0;
  }
}

The rest of the changes were along those lines for the major portions of the page—override the placement settings for the various bits and pieces.

So now the blog should be readable on small devices.

I hope.

---

Driskell v. Homosexuals

It seems that one Sylvia Ann Driskell is suing homosexuals (link via Flutterby). The handwritten lawsuit is a riot to read, but ultimately, it does seem that Ms. Driskell might be in need of some mental care (if not a proofreader).

Also, I think that Ms. Driskell needs to read (or listen) to Matthew Vines' talk on the Bible and homosexuality (it's long, but I think it's worth the time if only for some Christians to gain some perspective, and for gays to get some counter arguments for the Westboro Baptist Churches out there).

Wednesday, May 06, 2015

Dogs and cats living together

I'm watching an animated interview of Buckminster Fuller when I see this sequence of equations:

a = b a² = ab a² - b² = ab - b² (a+b)(a-b) = b(a-b) a+b = b 2b = b 2 = 1

And I'm thinking, that looks right, as far as I can remember algebra, but two can't be equal to one, can it?

I had to work through it by hand to find the problem, and now, gentle reader, you get to work through it yourself.

You're welcome.

Thursday, May 07, 2015

Small websites need not apply

Hoade and I were exchanging emails about AdWords today. He's trying to get his writing career into overdrive, seeing how every other job he's tried never worked out. I advised him against AdWords unless someone else was footing the bill, as it can get expensive.

I've also heard first hand that AdWords wasn't worth the money spent on it.

On the flip side, I found that AdSense wasn't worth it for me. Sure, if my blog had a bit more focus, like I was writing about gendered bodies in Japanese pornographic anime and horror through a Foucauldian framwork in order to analyze the West's gaze upon the world, Google would have had a better time generating related advertising towards the blog, but alas, my blog isn't that focused and the ads I did get were bizarre. And it didn't help me that not many people were seeing the ads.

Things might have changed since my last experiment with AdSense (or what I heard about AdWords), but I still cautioned Hoade about wasting money on AdWords. I feel he would do better by getting his name out there on podcasts and web-based forums. At least that way, he won't spend any money.

Last words from Hoade:

Just to add a little coda to our AdWords / Facebook ads convo, the for-real serious consensus seems to be "Maybe it could work. Do some A/b testing with your massive pile of ad money. One way might could work gooder than another." Not even XXXXXXX kidding.

Actual quote: "She ended up with a 1.3 percent click-through rate, which is actually very good." I believe this is true, but lord, it doesn't make you run for your checkbook.

Yeah, I'm thinking small sites need not apply.

Friday, May 08, 2015

I, for one, welcome our new self-driving robotic overlords

As I thought about Google's self-driving car, I realized that more than just taxicab drivers should be worried—there would be tremendous pressure from the trucking industry (and I'm not talking about the drivers here) to allow driverless trucks on the road. I'm not terribly surprised.

License plates are rarely an object of attention, but this one’s special the funky number is the giveaway. That’s why Daimler bigwig Wolfgang Bernhard and Nevada governor Brian Sandoval are sharing a stage, mugging for the phalanx of cameras, together holding the metal rectangle that will, in just a minute, be slapped onto the world’s first officially recognized self-driving truck.

The truck in question is the Freightliner Inspiration, a teched-up version of the Daimler 18-wheeler sold around the world. And according to Daimler, which owns Mercedes-Benz, it will make long-haul road transportation safer, cheaper, and better for the planet.

“There’s a clear need for this generation of trucks, and we’re the pioneers who are willing to tackle it,” says Bernhard.

Via InstaPundit, The World's First Self-Driving Semi-Truck Hits the Road | WIRED

I am also reminded of Humans Need Not Apply, but it might be all that dire.

Saturday, May 09, 2015

The Dymaxion Car

The Dymaxion Car, designed by Buckminster Fuller. It wasn't pretty. It could seat eleven. It was difficult to drive. It killed one of its first drivers. But it got 30mpg and could travel at 90mph. Kind of a mixed bag for a car built in 1933.

Still, it might be somewhat fun to ride in it, as long as there's a trained driver.

Sunday, May 10, 2015

Eight years of greylisting

I've been hacking on my greylist daemon over the past few days. I'm not sure what, exactly, prompted me to start hacking away at it though. The last code change was in December of 2011—all code changes since then have been tweaks to the Makefile (the file that describes how to build the program). As I'm hacking on it, I've come to hate the code handing the protocol the components use for communications (there's the main component that manages the data and logic; there's the component that interfaces with sendmail and another one that interfaces with postfix).

And over the past few days, I've reflected over what I would do differently if I were to write the greylist daemon now and how well my decisions eight years ago held up.

One decision I made eight years ago was to write my own “key/value” store instead of using a pre-existing one. I rejected outright the use of an SQL database engine (like MySQL or PostgreSQL) and I don't think I would change my mind now. The data stored is short lived (six hours for most entries, otherwise thirty-six days) and I don't think such churn is good for database engines.

In addition, the only NoSQL based solution (as they're now called) at the time was memcached (written in 2003; redis wasn't released until 2009, two years after I released the greylist daemon). memcached (and redis) can expire entries automatically, and it could handle five out of the six lookups the greylist daemon makes. The one lookup neither one can handle (as far as I can tell) is the IP address lookup.

This lookup compares the IP address of the sending SMTP server against a list. The list describes an address range and what to do if the given IP address “matches.” For example:

         0.0.0.0/0 GREYLIST
 205.211.164.50/32 ACCEPT
   206.214.64.0/19 REJECT
   207.115.11.0/26 ACCEPT

If, say, the IP address is 207.115.11.8, then the email is accepted and further processing is skipped because of the matching rule:

   207.115.11.0/26 ACCEPT

An IP address of 206.214.64.10 is rejected, because of the matching rule:

   206.214.64.0/19 REJECT

An address like 66.252.224.242 will match the rule

         0.0.0.0/0 GREYLIST

and because the result is GREYLIST, futher checks are made.

There does not appear to be a way of handling this type of query using memcached or redis. I would have to write code to store the IP addresses anyway. Also, memcached is a pure memory cache—if it crashes, all the data goes away (and remember—at the time I wrote this, this was really the only key/value store that existed that wasn't an SQL database engine) which is something I didn't want to happen. So my decision at the time to write my own key/value store wasn't a bad one.

Today?

Today I might consider using redis to store what I could, but it's another component that, if it isn't available, I can't greylist incoming email (I have to allow the email in—fail safe and all that). Also, the code I wrote to store the non-IP address data was easy to write. I dunno. It's hard to say how I would store the data today.

The protocol between my components is something I would handle completely differently today. I can't say what the actual protocol would be though.

There are basically two methods of sending data—a series of values in a fixed order (which is how the protocol works today) or as a series of tagged values, which can appear in any order. The former doesn't really deal well with optional data (you end up tagging such values anyway) while the later is harder to parse (since the values aren't in a fixed order, you have to deal with missing values in addition to duplicate values).

The biggest issue I have with the protocol now is what I said above, the code that handles the protocol is a mess—it's all over the place instead of in a few isolated routines. That makes updating the protocol (say, adding new fields, fixed or optional) very difficult.

What I would do now is make the protocol handing portion more of a module—a module for version 1.0 of the protocol, a module for version 1.1 of the protocol, etc., load them all up, and based upon the version embedded in the packet (something I do have, by the way), farm out the processing to the proper protocol version module. It would make updating the protocol easier to deal with in the codebase. The lack of this approach to the protocol is, I think, the biggest problem with the codebase today.

One last aspect I would change is the logging of verious statistics, or “key performance indicators” as they are called in the industry. Instead of incrementing a bunch of variables in the codebase and every so often dumping them out to syslog (messy code requiring the use of signals and all the problems that entail, and several lines of code modified for every new KPI added) I would use the method they use at Etsy—statsd—or at least, my own take on it. I don't need the full blown “all-singing, all-dancing, all-graphing” statsd that Etsy developed but one that just logs to syslog(). And given the whole concept is easy, a small version that just logs to syslog() is pretty trivial to write (I wrote a version in Lua with 225 lines of code, and a full quarter of that is just parsing the command line). The nice thing about a statsd-like concept is that it is trivial to add new KPIs to the codebase, and they're logged automatically without any other changes. The logging and potentially resetting of values is all isolated in statsd, in the way that log messages are logged to files or forwarded to another server is isolated in syslog.

There's not anything else I would really modify in the greylist daemon. Really, the only bad decision I made eight years ago was not fully isolating the protocol. Everything else was an okay decision.

And frankly, I'm not even sure if the greylist daemon needs any more work done on it.

Monday, May 11, 2015

SPF might not be worth handling, but what about RBL?

A month ago, I re-evaluated the use of SPF as an anti-spam measure and found it wanting. Today, I decided to re-evaluate my stance on the various real-time blackhole lists that exist. I was relunctant to use an RBL because of over-aggresive classification for even the smallest of infractions could lead to false positives (wanted email being rejected as spam). It has been over a decade since I first rejected the idea, and I was curious to see just how it would all shake out.

I used the Wikipedia list of RBLs as a starting point, figuring it would be pretty up-to-date. I then dumped information from my greylist daemon. The idea is to see how much additional spam would be caught if, after getting a “GO!” from the greylist daemon, I do a RBL check.

Out of the current 2,830 entries, only 145 had not been whitelisted. I didn't filter these out before running the test, but I don't think it would throw off the results too much. Half an hour of coding later, and I had a simple script to query the various RBLs for each unique IP address (1,446). I let it run for a few hours, as it had quite a few queries to make (1,446 IP addresses, each one requiring one query to see if the IP address is a known spammer, and a possible second one for the reason, across 45 RBL servers—it took awhile).

First up, how many “spam” results did I get from each RBL:

Results from each RBL

RBL	hits	reasons given
truncate.gbudb.net.	108	108
dnsbl.proxybl.org.	0	0
dnsbl-1.uceprotect.net.	132	132
dnsbl-2.uceprotect.net.	145	145
dnsbl-3.uceprotect.net.	23	23
dnsbl.sorbs.net.	65	65
safe.dnsbl.sorbs.net.	65	65
http.dnsbl.sorbs.net.	0	0
socks.dnsbl.sorbs.net.	0	0
misc.dnsbl.sorbs.net.	0	0
smtp.dnsbl.sorbs.net.	0	0
web.dnsbl.sorbs.net.	21	21
new.spam.dnsbl.sorbs.net.	37	37
recent.spam.dnsbl.sorbs.net.	184	184
old.spam.dbsbl.sorbs.net.	0	0
spam.dbsbl.sorbs.net.	0	0
escalations.dbsbl.sorbs.net.	0	0
block.dnsbl.sorbs.net.	0	0
zombie.dbsbl.sorbs.net.	0	0
dui.dnsbl.sorbs.net.	0	0
rhsbl.sorbs.net.	0	0
badconf.rhsbl.sorbs.net.	0	0
nomail.rhsbl.sorbs.net.	0	0
sbl.spamhaus.org.	293	293
xbl.spamhaus.org.	53	53
pbf.spamhaus.org.	0	0
cbl.abuseat.org.	36	37
psbi.surriel.com.	0	0
intercept.datapacket.net.	186	186
db.wpbi.info.	0	0
bl.spamcop.net.	65	65
noptr.spamrats.com.	224	224
dyna.spamrats.com.	208	208
spam.spamrats.com.	15	15
bl.spamcannibal.org.	96	96
spamtrap.drbl.drand.net.	0	0
blacklist.hostkama.com.	0	0
dnsbl.dronebl.org.	2	2
list.quorum.to.	1309	1309
ix.dnsbl.manitu.net.	48	48
dnsbl.inps.de.	627	627
bl.blocklist.de.	6	6
srnblack.surgate.net.	21	21
all.s5h.net.	363	363
rbl.megarbl.net.	54	54

As you can see, some of them were not worth querying. Also, about list.quorum.to … it's not straightforward to use that server as it always sent back a result even when the others did not. I ultimately decided that any result that only had a “hit” from list.quorum.to to be “non-spam” because of the issues.

I then proceeded to pour through all 2,830 results.

Email classification from RBLs

Marked as SPAM	2739	97%
Not marked as SPAM	91	3%
Total	2830	100%

And out of the 91 that was not marked as spam, only 7 were spam not marked by any of the RBLs. Not bad. But the real test is false positives—email marked as spam that isn't. And unfortunately, there were a few:

False positives

bounce.twitter.com	10
icpbounce.com	2
bounce.linkedin.com	3
returns.groups.yahoo.com	8
facebookmail.com	6
Total	29

Now, I realize that some of my readers might very well consider email from Twitter or Facebook as spam, but hey, don't judge me!

Ahem.

Anyway, that's a problem for me. I will occasionally have issues with the greylisting in some cases (rare, but it does happen, and I have to explicitely authorize the email when I become aware of the issue) but it's even worse with this. For instance:

66.220.155.148	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.151	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.143	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.136	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	FAIL	ix.dnsbl.manitu.net.
66.220.155.172	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.140	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.142	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	FAIL	intercept.datapacket.net.	ix.dnsbl.manitu.net.
66.220.155.144	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	FAIL	ix.dnsbl.manitu.net.
66.220.155.137	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	FAIL	recent.spam.dnsbl.sorbs.net.	ix.dnsbl.manitu.net.
66.220.155.141	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	FAIL	dnsbl.dronebl.org.	ix.dnsbl.manitu.net.
66.220.155.147	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.152	notification+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.152	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!
66.220.155.150	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	FAIL	ix.dnsbl.manitu.net.
66.220.155.138	update+iedcilif@facebookmail.com	XXXXXXXXXXXXX@conman.org	GO!

It's hit-or-miss within the IP range Facebook uses to send email. This would make troubleshooting quite difficult. I could whitelist the problematic domains but for any new site I might want to receive email from, I would have to watch the logs very closely for issues like this. But it's not as bad as I thought it would be, and it would cut out a lot of the spam I do get. It's tempting.

I shall have to think about this.

Tuesday, May 12, 2015

He is the electric messiah! The AC/DC god!

Today we received an email from the Marketing Department of the Corporation Overlord Corporation touting their latest press release to the public. They linked to the press release at the various GoogleMyFacePlusTwitterSpaceBook sites using some outrageous graphics. The one for Facebook was pretty scary looking, being based on this:

I'm not sure why the Marketing Departmemt of the Corporation Overlord Corporation felt the standard linking images for FaceTwitterGoogleMyPlusSpaceBook weren't good enough and needed to “kick it up a notch,” but there you go.

But as a counter point to that (or maybe even a counter-counter point, or a point, or something), here's an interesting video (warning: it's an hour) on the digital tracking that MyFaceGoogleSpaceBookPlusTwitter can (and most likely, is doing), if you can stomache the whole “viva la revolución my democratic comrades” vibe the speaker gives off (and you can pretty much skip the last fifteen to twenty minutes where it gets really thick).

Wednesday, May 13, 2015

Notes on an overheard conversation at The Ft. Lauderdale Office of The Corporation

“Can you help me?”

“Sure. What's the problem?”

“When I'm logged into my laptop as me, I can run ispell. But if I switch to root, it's not there.”

“Hmm … where does ispell live?”

“It's in /usr/local/bin. And before you ask—I checked, /usr/local/bin is in root's $PATH.”

“Hmmm … I'm running … um … that version of the operating system.”

“Rabid Wombat? I am too.”

“Let me drive the laptop for a second.”

“Okay. These two terminal windows.”

“Can you switch to root for me?”

“There you go.”

“Now, let's see … as you, I can see ispell in /usr/local/bin and the permissions seem okay. Now in the root terminal window … wow! There's a completely different set of files in /usr/local/bin. Hmmm … ”

“Any ideas?”

“Wait a second … that root window is on another system!”

“Oh … that would explain my problem … ”

“Yes it would.”

Thursday, May 14, 2015

The Flying Camera

I have a soft spot for cameras. I've got a few 35mm cameras, a couple of 8mm cameras and floating around here somewhere is a Super-8 camera. I also have … um … I think three, maybe four, digital cameras (which includes the one in my iPhone).

So yeah, I like cameras.

And even with my current crop of seldom-used cameras, I want the Lily (link via MyFaceTwitterGoogleSpaceBookPlus). The camera you toss in the air and it follows you around.

It's probably for the best that you can only pre-order the thing for now. And I live less than a mile from an airport.

Sigh.

Friday, May 15, 2015

Countrly Road

For no good reason, here's a video of some Japanese musicians doing a cover of John Denver's “Take Me Home, Country Raod” (link via Instapundit).

Saturday, May 16, 2015

The more things change

So.

HTTP2 is finally here (link via Hacker News). I'm not happy about it, but what can I (or you) do? It's a done deal.

Part of the reason I don't like it is that it seems as if Google pushed this for their own needs.

You have a completely warped perspective here.

This is something Google pushed, so that Google can have as many tracking cookies as they like when you browse the internet, without the cookies causing a noticeable performance degradation because a http request might exceed the American DSLs MTU size.

This was one of the primary engineering criterias. No really.

There's no features in it for the user.

You have a completely warped perspective here. This is something Google pushed,… | Hacker News

Google is now in a position to dictate the architecture of the web. Sure, one could ignore Google and blithely go about their web business, but really, if you want to even have a chance of being found on the web, you follow the dictates commands of Google! Heck, even I kept mucking with my blog until I got the “okay” from Google (although there were other reasons I did the change besides Google, notice I didn't stop until Google said I was okay). And don't think Google will stop there (which is another rant for another time).

Another reason I don't like HTTP2 is that, as written, it's TCP over TCP. I can understand why they did it that way, but it's sad that for as much power as Google has, even they couldn't force a more sensible change.

Sigh.

Plus ça change, plus ils deviennent énervant.

Sunday, May 17, 2015

The blind men and the Molochian elephant

Bostrom makes an offhanded reference of the possibility of a dictatorless dystopia, one that every single citizen including the leadership hates but which nevertheless endures unconquered. It’s easy enough to imagine such a state. Imagine a country with two rules: first, every person must spend eight hours a day giving themselves strong electric shocks. Second, if anyone fails to follow a rule (including this one), or speaks out against it, or fails to enforce it, all citizens must unite to kill that person. Suppose these rules were well-enough established by tradition that everyone expected them to be enforced.

So you shock yourself for eight hours a day, because you know if you don’t everyone else will kill you, because if you don’t, everyone else will kill them, and so on. Every single citizen hates the system, but for lack of a good coordination mechanism it endures. From a god’s-eye-view, we can optimize the system to “everyone agrees to stop doing this at once”, but no one within the system is able to effect the transition without great risk to themselves.

And okay, this example is kind of contrived. So let’s run through – let’s say ten – real world examples of similar multipolar traps to really hammer in how important this is.

Meditations On Moloch | Slate Star Codex

I don't agree with everything said in this long article (and warning—it is long. I mean, long. Did I mention just how long it was?) but I do feel that certain someones would benefit greatly if they read it and thought long and hard about it. While I'm tempted to give my summary of the article I'd rather not, lest my intended target audience disreguard the article entirely.

Monday, May 18, 2015

Gaming the system

By December 2011, lajello’s profile had become one of the most popular on the entire social network. It had received more than 66,000 visits as well as 2435 messages from more than 1200 different people. In terms of the number of different message received, a well-known writer was the most popular on this network but lajello was second.

How a Simple Spambot Became the Second Most Powerful Member of an Italian Social Network | MIT Technology Review

Only lajello isn't a human, but a spambot, but using the information in the article to boost your own ranking on MyFaceGoogleSpaceBookPlusTwitter is left as an exercise for the reader.

Tuesday, May 19, 2015

Wishful thinking

There are times when I wish the RFCs had more examples that covered various corner cases, such as handling SMTP or even, you know, SIP!

Ahem.

Anyway, while I'm here, let me also ask for a way to log everything but only when something is going to fail and skip the logging for stuff that won't fail (that just wastes space). How hard can that be?

Wednesday, May 20, 2015

If you think signal handling in C sucks, try Lua

So I have this Lua module to handle signals I wrote …

Originally, I just set a flag that had to be manually checked, as that was the safest thing to do (make that “practically the only thing you can do” if you want to be pedantic about it).

But after a while, I thought it would be nice to write a handler in Lua and not have to manually check a flag. Unfortunately, signal handlers have to be thought of as asynchronous threads that run at the worst possible time, which means calling into Lua from a signal handler is … not a good idea. The way to handle this is to hook into the Lua VM in the signal handler. lua_sethook() is the only safe Lua function to call from an asynchronous thread (or signal handler) as it just sets a flag in the Lua VM. Then when the Lua VM is at a safe spot, the hook is call which can then run the Lua function.

So we write our Lua function:

function Lua_signal_handler()
  print("Hi!  I'm handing a signal!")
end

and here I'm making it a global function just for illustrative purposes. At some point, we catch the signal to install the signal handler (which has to be in C):

/*------------------------------------------------------------------------
; In order to hook the Lua VM, we need to statically store the Lua state. 
; That's what this variable is here for ...
;-----------------------------------------------------------------------*/

static lua_State *gL;

	/* ... code code blah blah ... */

	/*----------------------------------------------------------------
	; we're in some function and we have access to a Lua state.  Store
	; it so the signal handler can reference it.
	;----------------------------------------------------------------*/

	gL = L;

	/*---------------------------------------------------------------
	; sigaction() should be used, but that's quite a bit of overhead
	; just to make a point.  Anyway, we are installing our signal
	; handler.
	;--------------------------------------------------------------*/

	signal(SIGINT,C_signal_handler);

The signal handler installs a hook:

static void C_signal_handler(int sig)
{
  lua_sethook(gL,luasignalhook,LUA_MASKCALL | LUA_MASKRET | LUA_MASKCOUNT,1);
}

and when it's safe for the VM to call the hook, it does, which then calls our signal handler written in Lua:

static void luasignalhook(lua_State *L,lua_Debug *ar)
{
  /*------------------------------------------------------------------
  ; remove the hook as we don't want to be called over and over again
  ;------------------------------------------------------------------*/

  lua_sethook(L,NULL,0,0);

  /*--------------------------------------------------------------------------
  ; get our function (which is a global, just for illustrative purposes, and
  ; call it.
  ;--------------------------------------------------------------------------*/

  lua_getglobal(L,"Lua_signal_handler");
  lua_call(L,1,0);
}

Yes, it's a rather round-about way to handle a signal, but that's what is required to run Lua code as a signal handler. And it works except for two cases (that I have so far identified—there might be more).

The first case—coroutines. Lua coroutines can be thought of as threads, but unlike system threads, they have to be scheduled manually. And like system threads, signals and coroutines don't mix. Each coroutine creates a new Lua state, which means that if a signal happens, the Lua state that is hooked may not be the one that is currently running and thus, the Lua-written signal handler may never be called!

The second issue involves a feature of POSIX signals—the ability to restart system calls. Normally, a signal will interrupt a system call and its up to the program to restart it. There is an option to restart a system call automatically when a signal happens so the program doesn't have to deal with it. The funny thing is—under the right conditions, the Lua signal handler is never called! Say the program is making a long system call, such as waiting for a network packet to arrive. A signal is raised, our signal handler is called, which hooks the Lua VM. Then the system call is resumed. Until a packet arrives (and thus we return from the system call) the Lua VM never gets control, and thus, our function to handle the signal is never called (or called way past when it should have been called).

Fortunately, I found these issues out in testing, not in production code. But it has me thinking that I should probably work to avoid using signals if at all possible.

Thursday, May 21, 2015

Notes about an overheard conversation at The Ft. Lauderdale Office of The Corporation

“We should definitely do that at Black Hat!”

“I didn't know you were into haberdashery.”

“What?”

“I think you mean millinery. Haberdashers generally sell buttons and thread and stuff.”

“Really?”

“Yes.”

“What are you guys talking about?”

“Your fascination with hats.”

“You mean Black Hat?”

“Yeah. Haberdashery.”

“Millinery!”

“Oh, sorry. Millinery.”

“You guys are crazy.”

“We're just pushing it to eleven.”

---

The check that was in the mail

I'm checking my snail mail and … what's this? A check?

You mean the check was in the mail?

Oh wait … it's one of those “promotional checks” and not a “real check,” even though it has a check number, it's made out to me, has the value as both numbers and words, it's signed, and has what look to be a routing number (to an Australian bank‽) and account number. Also, across the bottom it has: “THIS DOCUMENT CONTAINS A BLUE BACKGROUND, [Check! –Sean] MICROPRINTING [Oh yes, it does. It doesn't make much sense, but I can make out the letters. —Sean] AND AN ARTIFICIAL WATERMARK ON THE BACK [Yup, “IPS.” So, check! —Sean] — VOID IF NOT PRESENT” [Nope, it's all there, so it's not void. —Sean]

And there is the story of man who deposited a “promotional check” for $95,000 …

So maybe it's a real check?

Perhaps I could try depositing it? [No! —Bunny] [Awwww! —Sean] [Okay, it's your bank account to lose … go right ahead! —Bunny] [Woot? —Sean]

Friday, May 22, 2015

“Consistent mediocrity, delivered on a large scale, is much more profitable than anything on a small scale, no matter how efficient it might be.”

Fundamentally, there’s a theme in Olia’s speech (and the speech of others in that space, like Dragan Espenschied, Ben Fino-Radin, and so on) bemoaning the move away from a space on a website being the province of the users, and being turned into a homogenized, commodified breeder farm of similar-looking websites with only surface implementations, like WordPress, Facebook Pages, and so on.

…

There was a time when a person who was not particularly technical, or whose technical acumen was sufficient to get applications running on a machine and not much more, could code a webpage. The tags were pretty straightforward, the uses of them clear, and the behavior pretty dependable. Much how one could, in a weekend, learn sufficiently how to pilot a sailboat… such was that a few weekends of study could allow a person to craft a fun little webpage, with their voice, their stamp, and the idiosyncrasies of their personality shining through.

Those days are gone. Long gone.

Instead, we have this (as my buddy Ted Nelson calls it) nightmare honky- tonk of interloping, shifting standards soirees that ensure, step by step, bylaw by beta, that anybody who isn’t willing to go full native will be shut out forever. The Web’s underpinnings, at least on the basic HTML level, have been given over to the wonks and the engineers, making it an impenetrable layer of abstraction, not worth your time to learn unless you were looking to buff up your resume, or if some programmer pride resided in this whole mess being in your job description.

That Whole Thing With Sound in In-Browser Emulation « ASCII by Jason Scott

There's no need to read the full article (unless you are interested in the state of audio in webpages and how it's not serving web based emulators of old home computers letting people play thousands of games from the 80s and 90s); I'm just quoting the part that spoke to me.

I'm also reminded of this:

For a very long time, taste and artistic training have been things that only a small number of people have been able to develop. Only a few people could afford to participate in the production of many types of media. Raw materials like pigments were expensive; same with tools like printing presses; even as late as 1963 it cost Charles Peignot over $600,000 to create and cut a single font family.

The small number of people who had access to these tools and resources created rules about what was good taste or bad taste. These designers started giving each other awards and the rules they followed became even more specific. All sorts of stuff about grids and sizes and color combinations — lots of stuff that the consumers of this media never consciously noticed. Over the last 20 years, however, the cost of tools related to the authorship of media has plummeted. For very little money, anyone can create and distribute things like newsletters, or videos, or bad ass tunes about "ugly."

Suddenly consumers are learning the language of these authorship tools. The fact that tons of people know names of fonts like Helvetica is weird! And when people start learning something new, they perceive the world around them differently. If you start learning how to play the guitar, suddenly the guitar stands out in all the music you listen to. For example, throughout most of the history of movies, the audience didn't really understand what a craft editing was. Now, as more and more people have access to things like iMovie, they begin to understand the manipulative power of editing. Watching reality TV almost becomes like a game as you try to second-guess how the editor is trying to manipulate you.

As people start learning and experimenting with these languages authorship, they don't necessarily follow the rules of good taste. This scares the shit out of designers.

In Myspace, millions of people have opted out of pre-made templates that "work" in exchange for ugly. Ugly when compared to pre-existing notions of taste is a bummer. But ugly as a representation of mass experimentation and learning is pretty damn cool.

Regardless of what you might think, the actions you take to make your Myspace page ugly are pretty sophisticated. Over time as consumer-created media engulfs the other kind, it's possible that completely new norms develop around the notions of talent and artistic ability.

Happy Ugly.

the show: 07-14-06 - zefrank

But sadly, no one cares.

Saturday, May 23, 2015

Sometimes, you just gotta go back to the 8-bit era

In the pages that will follow, I will be documenting the various stages in the design of a new arcade game that I hope to create for my classic Tandy Color Computer 3 sold internationally by the Radio Shack Corporation during the 80's and early 90's. This game will largely be created the old school way utilizing as much as possible the same setup that I used to develop games back then.

[PopStar Pilot]

As a teenager with a computer during the 80s I always had the idea of writing a computer game in the back of my mind, but I never did know how to write one. It perhaps didn't help that I had a Tandy Color Computer at the time. I know, it's a bad craftsman that blames his tools, but in this case, I think there's something to it. The Color Computer had no hardware graphics to speak of (the Color Computer 3 did, but I had moved on to the PC world by the time it came out) so it was up to the programming to do all the bit shifting, masking and drawing which isn't as easy as it sounds (or rather, making it fast isn't that easy).

I never did write a game.

But it is a simple computer. Unlike modern systems, the entire computer is documented in a 70-page book and games were written for it. So feeling a bit nostalgic, I fired up an emulator (I'm nostalgic, not masochistic) and spent a few hours getting a simple graphic program going.

Yeah, not that easy. That running man? (bonus points if you recognize where he comes from) There're eight images in the animation, and each image is repeated four times, each one shifted right one pixel to avoid having to do a massive amount of shifting at runtime (it's a classic “memory vs. time” tradeoff here). Then I had to align the images so it looks smooth (image one, then image two shifted right one pixel, then image three shifted right two pixels, then image four shifted right three pixels and that takes us through a full byte of pixels) which complicated the animation loop since it ends with image one shifted one pixel to the right, which has to carry over to the next loop (image one shifted right one pixel, then image two shifted right two pixels, then image three shifted right three pixels, then image four not shifted but starting one byte over, etc).

That's not to mention that I had to draw the running man over the background image which requires merging the image data of the man with the background image. And to avoid really weird drawing artifacts, I used a double-buffer method (show one frame while drawing into a non-visible frame, then show the updated frame and use the previous frame to draw and repeat).

It was fun though. I don't think this will end up as a game any time soon, but it was nice to work on a computer that is so easily comprehensible by one person and where hitting the hardware is very easy to do (I think the last time I programmed to the hardware was in the mid-90s). I think my nostalgia has been sated for now.

Sunday, May 24, 2015

The crazy things that were done to make games run fast in the 8-bit era

The most amout of time spent in my simple graphics program is this loop:

clrloop		ldd	,u++	; load 2 bytes from background, increment pointer to next 2 bytes
		std	,x++	; store them in current frame, increment pointer to next 2 bytes
		cmpu	#end	; are we done yet?
		bne	clrloop	; nope, keep going

Despite its name, it's not clearing memory. It's actually copying the background image to the current frame being drawn. I'm not showing the code before or after this as this post is really about this loop.

As written, each iteration of this loop takes 24 clock cycles (or just “cycles”) to run, meaning this code effectively copies one byte every 12 cycles. I recalled reading several years ago a crazy scheme to copy memory on the Motorola 6809 (the CPU used in the Color Computer) that involved using the stack register.

But before I get crazy, just how fast can I get the code to run?

Unrolling the loop a bit:

clrloop        ldd     ,u++	; load 2 bytes from background
               std     ,x++	; store 2 bytes to current frame
               ldd     ,u++	; repeat this seven more times
               std     ,x++
               ldd     ,u++
               std     ,x++
               ldd     ,u++
               std     ,x++
               ldd     ,u++
               std     ,x++
               ldd     ,u++
               std     ,x++
               ldd     ,u++ 
               std     ,x++ 
               ldd     ,u++ 
               std     ,x++ 
               cmpu    #end 	; are we there yet?
               bne     clrloop	; don't make me turn this CPU around!

and we get 8.5 cycles per byte. Unrolling it more isn't worth it, as the fastest we'll get is 8 cycles per byte (assuming we unroll the entire loop to copy all 1,024 bytes; but in doing so we'll use 4K in code (ldd ,u++ and std ,x++ are both two byte instructions) just to copy 1K in data). Can we do better?

In checking the timings of various index operations, amazingly enough, adding an offset instead of just incrmenting the pointers is faster. This:

clrloop         ldd     ,u	; load 2 bytes from background
                std     ,x	; store 2 bytes to current frame
                ldd     2,u	; load 2 more from background past the previous data
                std     2,x	; store them past the previous data
                ldd     4,u	; and keep this up
                std     4,x
                ldd     6,u
                std     6,x
                ldd     8,u
                std     8,x
                ldd     10,u
                std     10,x
                ldd     12,u
                std     12,x
                ldd     14,u
                std     14,x
                leau    16,u	; adjust pointers by 16 bytes
                leax    16,x	; as that's how much we copied
                cmpu    #end	; are we there yet?
                bne     clrloop ; enough!

gets us down to 6.5 cycles per byte! But unrolling this any further won't buy us a thing, as once the index passes 15, the instruction takes longer to execute because of additional instruction decoding. So this routine is pretty much it as far as a straightforward approach will take us. Not so bad though—almost twice as fast as the original 4 instruction loop. But to go even faster, we have to get crazy and bring in the stack pointer.

Why the stack pointer?

Because of four instructions: PSHS, PSHU, PULS PULU. The first instruction can save a number of registers onto the stack. But looking at it another way: it's an instruction that can write up to 12 bytes into memory. The second instruction is similar, but instead of using the stack register, it uses the U register (it's the “user stack pointer”). The data written goes from higher addresses to lower addresses (because traditionally, stacks grow downward in memory). The last two instructions to the reverse, restoring a number of regsters from memory, or, reading up to 12 bytes into registers.

But we can't use all the possible registers these instruction support. We can't use the program counter as that's rather important to executing the program (it contains the address of the currently executing instruction—overwriting that will cause the program to start running who knows what). We'll be using both stack registers so those are out. We could use the CC register, but part of its use is to control the CPU—setting random values could be interesting. Too interesting for me, so that's out (and it's only 8-bits—not like a great loss).

That still leaves us with four other registers we can use: X, Y, D (16-bit registers) and DP (an 8-bit register). So realistically, we can transfer up to seven bytes at a time, taking 12 cycles to read and 12 cycles to write, for a theoretical maximum of 3.4 cycles per byte!

Woot!

The problem with this method is that the stack pointer is used by the CPU to keep track of where it is in the program. Not only that, but when the CPU receives an interrupt (a signal that somehing has happened and needs to be handled now!) it saves what it is doing on the stack and handles the interrupt. And while on the 6809 the stack register can be used as a general purpose index register (like we've been using the X and U registers) we're hampered by the fact that interrupts happen.

In this case though, it can be done. The stack grows downward in memory—that is, as items are pushed onto the stack, the stack pointer is decremented lower and lower into memory. Taking this into account means we will be filling in the frame backwards, from the bottom of the frame towards the top. So we set the stack pointer to the bottom of the frame. If an interrupt happens, sure, there's some odd stuff written to the frame, but once the interrupt has been handled, the stack pointer is restored to were it was before the interrupt and we can continue copying the background, overwriting the garbage data added by the interrupt.

But pulling data off a stack goes from lower addresses to higher. And the bytes are pulled off in reverse order they were pushed (as expected—it's a stack after all—last in, first out). So the background data would need to be rearranged to take into account that we're reading the data from a low to high, storing the data high to low, every seven bytes are reversed, and that the memory in front of a frame can be expected to be trashed by an interrupt. Then there's the issue that 7 does not evenly divide 1,024—we'll have to handle the last two bytes as a special case.

But assuming the data is stored correctly and we have some memory in front of each frame that can be safely transhed, then the copying code would be:

clrloop		pulu	dp,d,y,x ; transfer seven bytes
		pshs	x,y,d,dp
		cmpu	#end-2   ; are we there yet?
		bne	clrloop  ; shut up!
		pulu	d	 ; some post loop cleanup
		pshs	d

and get 4.5 cycles per byte.

But this level of optimization should only be done if absolutely required. And in my case, it's not required (yet—if it ever will be). I'll be happy to stick with the 6.5 cycle version for now.

Monday, May 25, 2015

Oh that Florida!

Ironically, one of the things that may be contributing to Florida being shamed so often in the national media is something all Floridians should be proud of.

The terms "progressive" and "model for the rest of the nation" don't often appear in sentences with "Florida," but that's exactly how people view the state's open-records laws, AKA the Government in the Sunshine Act.

Since 1909, Florida has had a proud tradition that all government business is public business and therefore should be available to the public. That means all records, including photos and videos, produced by a public agency are easily accessible with a few narrow and obvious exceptions. Public officials are also required to open all of their meetings — even unofficial ones — to the public.

…

However, those same laws are also the reason your mugshot appears online days after your arrest, and those laws make it incredibly easy for journalists to write about weird Florida news stories.

You'll notice something when you read so many "Weird Florida" news stories. They almost always include the phrase "according to the arrest report."

As journalists, all we have to do in most cases is call the police department and ask for an arrest report, and the cops are required to give it to us. Nowadays a lot of cops simply email the reports, and some departments even post arrest records online. Some of the more dedicated weird-Florida-news reporters go through batches of arrest reports at a time.

How Florida's Proud Open Government Laws Lead to the Shame of "Florida Man" News Stories | Miami New Times

You know, that explains a lot. It's not that Florida is crazier than the rest of the nation, it's just that the rest of the nation has decided not to air its dirty laundry.

Or in other words, Florida is the most transparent state when it comes to governance. Go figure.

Tuesday, May 26, 2015

Notes on an overheard conversation at The Ft. Lauderdale Office of The Corporation

“Man, the new hires seem to be getting younger and younger every day. And unruly.”

“You do realize today is ‘Bring Your Kid To Work Day,’ don't you?”

“…”

“Yup.”

“Is it too late to call in si—”

“Yes.”

“Okay.”

---

Notes on another overheard conversation at The Ft. Lauderdale Office of The Corporation

“It's too quiet in here. The kids are up to something.”

“Now, now, be nice.”

“I'm tellin you, they're planning something … ”

“All the kids have gone home.”

“Yeah, right! That's want they want us to think.”

Wednesday, May 27, 2015

Der Ring des Star Wars

So, by now you’re probably wondering what any of this has to do with Star Wars?

Well, as this essay will show, the six Star Wars films together form a highly structured ring composition. The scheme is so carefully worked out by Lucas, so intricately organized, that it unifies the films with a common universal structure (or what film scholar David Bordwell might call a “new formal strategy”), creating a sense of overall balance and symmetry.

Via Sean Tevis on GoogleMyTwitterFaceSpaceBookPlus, star wars ring theory | Mike Klimo

It's long, but it's an interesting new theory about Star Wars. Sure, we've all heard about George Lucas borrowing heavily from The Hero With A Thousand Faces, but a ring composition? That's certainly a new take on things.

But if the Star Wars films comprise a ring composition, it's only coincidental, as The Secret History of Star Wars made clear: George Lucas was making it up as he went along.

Thursday, May 28, 2015

Kung Fury

It's set in the 80s. It's set in Miami. It's a Kung Fu cop. It has Adolf Hitler (aka Kung Fürher). And it has a T-Rex. It's the ultimate in 80s actions films not made in the 80s. It's Kung Fury!

Oh lord is this thing over the top (so over the top it has its own music video staring David “The Hoff” Hasselhoff). It's more a series of quick scenes that ape common 80 action film tropes turned up to 11 than it is a film with a compelling story and character development. And all the more glorious because of it.

Friday, May 29, 2015

Does the removal of the audio from this video make it ironic?

This video previously contained a copyrighted audio track. Due to a claim by a copyright holder, the audio track has been muted.

Yeah, that happens on YouTube. But for some reason, the removal of the audio track on Anna's performance of “XXXX You” just makes it that much better, because she's “performing” it using sign langauge.

How appropriate.

Saturday, May 30, 2015

Magic is supposed to be … well, magical! Not scientific!

RPG magic systems can roughly be divided up into "fixed spell" and "freeform" mechanics. Fixed spell systems are often highly mechanistic, where the operation of each spell is exactly calculable. Freeform mechanics, on the other hand, call for the GM to judge the difficulty of a spell based on little information as well as a large degree of randomness.

Neither of these, however, is "mysterious". A mystery means that no pattern is obviously visible – but there is a hidden pattern. For a magic system to be mysterious, there must be hidden patterns which the magician character does not know at first, but which can with effort be discovered. In a game, this means that there must be either hidden variables or even hidden rules. An extreme of this would be that the GM secretly designs the magic system and only lets the player learn it a bit at a time (i.e. completely hidden rules). However, mystery can be injected by having hidden variables. i.e. How a PC's magic works depends on factors which are defined by GM, but which the player must deduce from other clues.

Via Hacker News, Breaking Out of Scientific Magic Systems

This is more observations about magic in role playing systems than it is a system to use to replace an existing magic system. The author is right that we (modern players) tend to be reductionist about magic systems because of modern science in today's world (I know I am a reductionist when it comes to magic systems as a player—I never did get a good grip on the magic system in Mage, probably the closest role playing system where “magic” is still mysterious and needs to be discovered because the system was so vague and contradictory) and maybe we need to loosen up a bit. I don't know … it sounds like it would be a lot of work for the GM, and possibly alienate the players.

Sunday, May 31, 2015

We wouldn't want anything to happen to the page rank on your nice website, now would we?

For these reasons, over the past few months we’ve been running tests taking into account whether sites use secure, encrypted connections as a signal in our search ranking algorithms. We've seen positive results, so we're starting to use HTTPS as a ranking signal. For now it's only a very lightweight signal — affecting fewer than 1% of global queries, and carrying less weight than other signals such as high-quality content — while we give webmasters time to switch to HTTPS. But over time, we may decide to strengthen it, because we’d like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web.

Via Rob Landley's Blog Thing for 2015, Official Google Webmaster Central Blog: HTTPS as a ranking signal

And that was nine months ago. Is your website served over HTTPS?

This just appears to be yet more proof that Google is calling the shots on the web now.

Oh, by the way, your web server is HTTP/2 compliant, right? Wouldn't want anything bad to happen to your page rank, now would you?

• Current
• First
• Last
• Top
• Home
• About
• Archive
• Search
• Glossary
• Copyright
• Help
• Accessibility

Obligatory Picture

Obligatory Contact Info

• Comments? sean@conman.org

Obligatory Feeds

• RSS Feed
• Atom Feed
• JSON Feed

Obligatory Links

• Flutterby!
• KIRK.is

Obligatory Miscellaneous

• About the Source Code
• Source Code

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.