The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Monday, February 19, 2018

Why one might want to load dynamic objects from memory instead of from disk

Years ago I started on a little project entitled the LEM Project. It was a project to make distribution of applications written in Lua easier (It's named after the Apollo Lunar Moduleaka the Lunar Excursion Module; Lua is “moon” in Portuguese). I had a few goals:

  1. allow distribution of multiple versions of a Lua module;
  2. allow for different architectures for the same module written in C;
  3. run the entire application from the distribution file (nothing to install).

And I almost hit all the goals.

The first two were easy. I read over the ZIP file specification and saw that

  1. multiple files of the same name were entirely possible—nothing in the specification disallows it;
  2. additional metadata can be added to each file beyond what is defined in the specification.

So armed, I created zip file with some addtional metadata and …

[spc]lucy:~/projects/LEM>./zipf.lua sample.lem 
The Eagle Has Landed
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.env
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.errno
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.1    MODULES/org.conman.iconv
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.sys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.env
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.errno
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys.magic
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
Linux     x86       LGPL3+       Lua 5.1 5.1 1.1.1  MODULES/org.conman.iconv
Linux     x86       LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net.ipacl
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
Linux     x86       LGPL3+       Lua 5.1 5.1 1.2.0  MODULES/org.conman.sys
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.2  MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.tcc
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.cc
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.date
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.debug
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.dns.resolv
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.getopt
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.string
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.table
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.unix
                    MIT          Lua 5.1 5.1 0.10   MODULES/lpeg
                    MIT          Lua 5.1 5.1 0.10   MODULES/re
Linux     x86       MIT          Lua 5.1 5.1 0.12   MODULES/lpeg
Linux     x86       MIT          Lua 5.2 5.2 0.12   MODULES/lpeg
                    MIT          Lua 5.1 5.2 0.12   MODULES/re
Linux     x86       MIT/X11      Lua 5.1 5.1 0.4.work3 MODULES/zlib
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket
Linux     x86       MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.core
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.ftp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.http
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.smtp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.tp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.url
                    MIT/X11      Lua 5.1 5.1 1.0.1  MODULES/ltn12
                                                    FILES/APPNOTE.TXT
                                                    FILES/COPYING
                                                    FILES/README
                                                    FILES/Miscellaneous Things About Nothing
[spc]lucy:~/projects/LEM>

You can see there are several copies of each module. The modules listed without a system (like Linux or SunOS) are written in Lua; the other ones are in C and both the system and architecture are listed. The license is included, along with the Lua verions the module will work for (it's “minimum version” and “maximum version”) as well as the version of the module (if known).

It's even a valid zip file:

[spc]lucy:~/projects/LEM>unzip -l sample.lem 
Archive:  sample.lem
The Eagle Has Landed
  Length     Date   Time    Name
 --------    ----   ----    ----
    25472  05-24-14 17:10   MODULES/org.conman.base64
    13448  05-24-14 17:10   MODULES/org.conman.crc
    12200  05-24-14 17:10   MODULES/org.conman.env
    18688  05-24-14 17:10   MODULES/org.conman.errno
    57032  05-24-14 17:10   MODULES/org.conman.fsys
    24952  05-24-14 17:10   MODULES/org.conman.hash
    17664  05-24-14 17:10   MODULES/org.conman.iconv
    17648  05-24-14 17:10   MODULES/org.conman.math
    77944  05-24-14 17:10   MODULES/org.conman.net
    26296  05-24-14 17:10   MODULES/org.conman.pollset
    88256  05-24-14 17:10   MODULES/org.conman.process
    37848  05-24-14 17:10   MODULES/org.conman.strcore
    15392  05-24-14 17:10   MODULES/org.conman.sys
    24312  05-24-14 17:10   MODULES/org.conman.syslog
    14175  06-12-14 22:04   MODULES/org.conman.base64
     8214  06-12-14 22:04   MODULES/org.conman.crc
     7193  06-12-14 22:04   MODULES/org.conman.env
    10690  06-12-14 22:04   MODULES/org.conman.errno
    31885  06-12-14 22:04   MODULES/org.conman.fsys
    15567  06-12-14 22:04   MODULES/org.conman.fsys.magic
    14067  06-12-14 22:04   MODULES/org.conman.hash
    10197  06-12-14 22:04   MODULES/org.conman.iconv
    10816  06-12-14 22:04   MODULES/org.conman.math
    43651  06-12-14 22:04   MODULES/org.conman.net
    25248  04-18-14 20:06   MODULES/org.conman.net.ipacl
    15914  06-12-14 22:04   MODULES/org.conman.pollset
    49607  06-12-14 22:04   MODULES/org.conman.process
    15666  06-12-14 22:04   MODULES/org.conman.strcore
     9763  06-12-14 22:04   MODULES/org.conman.sys
    13303  06-12-14 22:04   MODULES/org.conman.syslog
    21218  06-12-14 22:04   MODULES/org.conman.tcc
     5617  06-12-14 22:04   MODULES/org.conman.cc
     2500  06-12-14 22:04   MODULES/org.conman.date
     3829  06-12-14 22:04   MODULES/org.conman.debug
     2966  06-12-14 22:04   MODULES/org.conman.dns.resolv
     3260  06-12-14 22:04   MODULES/org.conman.getopt
     2464  06-12-14 22:04   MODULES/org.conman.string
     5243  06-12-14 22:04   MODULES/org.conman.table
     2732  06-12-14 22:04   MODULES/org.conman.unix
    40081  05-25-14 15:17   MODULES/lpeg
     6029  05-24-14 00:36   MODULES/re
    40045  05-28-14 15:24   MODULES/lpeg
    40045  05-28-14 15:24   MODULES/lpeg
     6286  05-28-14 15:24   MODULES/re
    19794  05-30-14 21:29   MODULES/zlib
     4451  05-28-14 14:52   MODULES/socket
    55449  05-28-14 14:52   MODULES/socket.core
     9243  05-28-14 14:52   MODULES/socket.ftp
    12330  05-28-14 14:52   MODULES/socket.http
     8074  05-28-14 14:52   MODULES/socket.smtp
     3612  05-28-14 14:52   MODULES/socket.tp
    11036  05-28-14 14:52   MODULES/socket.url
     8331  05-28-14 14:52   MODULES/ltn12
   161412  05-09-14 01:24   FILES/APPNOTE.TXT
The ZIP file format
     7651  05-25-14 18:53   FILES/COPYING
     9789  06-07-14 22:13   FILES/README
Much Ado About Nothing
     3946  05-10-99 23:00   FILES/Miscellaneous Things About Nothing
If you this this has significance, think otherwise
 --------                   -------
  1250541                   57 files
[spc]lucy:~/projects/LEM>

(Although if you try to unzip this file, the unzip program will ask you what you want to do with the duplicate files.)

The intent was that only those modules that match the Lua version, system (if a C-based Lua module) and architecture (again, C-base Lua module) match, load the module; otherwise, it would be ignored.

And it worked. For the most part.

I could load Lua modules written in Lua directly from the zip file with no problem. I even got LuaRocks to run completely from the zip file (with some hacks so it could load the configuration file). But I could not do that for the C-based Lua modules. For those, you had to write them out to disk to load them up. And because of that, I lost interest in continuing the project.

Until now. I just came across MojoELF (via Project: 2ine which was linked via Hacker News)—a project to load ELF binaries directly from memory (and those C-based Lua modules I have are ELF binaries), which is exactly what I wanted all those years ago.


Fun with banging moon rocks

Back in Debtember, I was notified of a potential problem with one of my Lua modules, speficically, the LuaRocks specfile of said module—it failed to compile properly on Mac OS-X.

I finally got around to it and fixed the issue, and while I was doing that, I figured I might as well fix some of the other Lua modules I have.

You can always tell when I do this because the version numbers accumulate in a rapid succession. My CBOR Lua module went from version 1.2.4 to 1.2.11 in the span of an hour or so. And it's all due to the picky little details.

As I mentioned a few months ago, I pull the version directly from git. But I also have to define the version in the Makefile in case someone downloads the code instead of cloning the repository. But there have been times when I updated the GitHub repository without updating the Makefile, or forgot to properly rename the LuaRocks specfile (probably as a sanity check, the filename needs to include the version number which is also specified inside the file). To handle those cases, I added a script to git to check for such things before pushing the update to GitHub.

So it should be easy, right?

Not quite.

One issue I forgot to check in the script was to ensure the LuaRocks specfile was syntactically correct. So there goes one version.

The Lua module is written with C99. So I specified c99 as the compiler in the LuaRocks specfile. Only the default options LuaRocks uses breaks on Mac OS-X with c99 as the compiler. The why is simple—Mac OS-X uses clang, which supports different options than GCC. But LuaRocks defaults to using GCC when compiling on Mac OS-X. That's fine, because there's a GCC wrapper on Mac OS-X that translates the options for GCC to clang, but only if the compiler you use is GCC. On Mac OS-X, c99 defaults to clang so of course it fails because clang receives options for GCC.

I can use the default compiler that LuaRocks defines on Mac OS-X and it compiles fine. But not on Linux, where GCC defaults to “not C99.” I finally figured out the magical incantation to fix that:

build =
{
  platforms =
  {
    windows =
    {
      type    = "builtin",
      modules = 
      {
        ['org.conman.cbor_c'] = 
        {
          sources = { 'cbor_c.c', 'dnf.c' },
          defines = { 'VERSION="' .. source.tag .. '"' },
        },

        ['org.conman.cbor']     = 'cbor.lua',
        ['org.conman.cbor_s']   = 'cbor_s.lua',
        ['org.conman.cbormisc'] = 'cbormisc.lua',
      }
    }
  },
  
  type = "make",
  build_variables =
  {
    CC      = "$(CC)",
    CFLAGS  = "$(CFLAGS) -DNDEBUG -I$(LUA_INCDIR)",
    LDFLAGS = "$(LIBFLAG)",
    LUA     = "$(LUA)",
  },
  
  install_variables =
  {
    LIBDIR = "$(LIBDIR)",
    LUADIR = "$(LUADIR)",
    LUA    = "$(LUA)",
  },
  
  platforms =
  {
    linux   = { build_variables = { CC = "gcc -std=c99" } },
    solaris = { build_varaibles = { CC = "c99"          } },
  }
}

And it worked fine. But it burned through some additional versions as I worked out the details. And then I found an issue—if you look closely, you'll notice that I have the patforms stanza twice. That means the second stanza replaces the first stanza and thus, I lost the Microsoft Windows compiling instructions.

So yet another version goes by as I merge the two platforms stanzas into a single block.

And then I forgot to update the version number in the LuaRocks specfile. It had the right name, but the wrong internal version.

So there's another version and an update the the checking script.

By now, you might be wondering why I don't bother testing this before updating the version number. And it's simple—I can't! The LuaRocks specfile contains this bit:

package = "org.conman.cbor"
version = "1.2.11-1"

source =
{
  url = "git://github.com/spc476/CBOR.git",
  tag = version:match "[^-]+"
}

To test it, LuaRocks will download the code from GitHub using the given tag value, which is based off the version number. But if I haven't checked in the code for that tag, LuaRocks will fail. So I have to check in the code with the proper tag before I can test the LuaRocks file.

Yes, I could do the code updates and afterwards fix the LuaRocks specfile, but the script that's run prior to pushing changes checks to see if the latest version is a “good version,” like 1.2.10. It will reject a version like 1.2.10-1-g5b75b4f (which is git's way of saying “one commit past the previous tagged version”). I added that check to ensure LuaRocks would get the proper version of everything—I would only push when I had everything ready for a new version.

Ah—unintended consequences.

We'll see if the next update goes smoother. One of these days I'll get this right.

Wednesday, February 14, 2018

Again, it was quite crowded out tonight. What is it with February 14th and everybody eating out? I'm wondering if it's some sort of holiday

As we did last year, Bunny and I had a wonderful dinner at Roots Italian Kitchen, only this time, we were able to skip the whole non-Euclidean parking garage for the Biltmore that for some reason lurks here in downtown Boca Raton. Once you know where to look, it's easily skipped.

We walked inside and Yuriy (the owner of the restaurant with an incredible memory) greeted us warmly and thanked us for coming back and dining with his establishment on St. Valentine's Day.

[This place is so fancy, even the tables are garnished!]

We were shown to our table, which was decorated with rose petals. Every table was decorated with rose petals. It was an interesting touch I had never seen before.

[Why yes, I'm still trying to destroy the cliché that is a full plate of beautiful food by making my own cliché of an empty plate for formerly beautiful food.]

The appetizer was okay.

Okay, who am I kidding? It was an incredible lobster-and-crabcake appetizer that was easily the best part of the entire meal. That's not to say the rest of the meal was bad—it wasn't. The Chicken Bella Risotto (grilled portobellos with chicken in a creamy risotto) was wonderful (we both had it) and dessert was worth the wait (Tiramisu for Bunny, and a brownie á la mode sans mode for me; sadly, the only misstep of the meal was the restaurant running out of vanilla ice cream, but still, that brownie was good).

But that lobster-and-crabcake appetizer … wow!

It was a good thing we parked a few blocks away (to avoid that non-Euclidean garage lurking in downtown Boca Raton)—we could use the walk after a meal like that.

Monday, February 12, 2018

Tortoise and hare racing

I never expected this to be a thing, but tortoise vs. hare, tortoise vs. hare and tortoise vs. hare. And each time, the result is the same.

No, really.


Tough love, or stultifying ossification? I don't know

There is “Postel's Law” in protocol design that states:

2.10. Robustness Principle

TCP implementations will follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others.

TRANSMISSION CONTROL PROTOCOL—DARPA INTERNET PROGRAM—PROTOCOL SPECIFICATION

But then there's this:

4.5 Robustness

A well-designed protocol is robust.

Robustness and efficiency are often at odds. For example, although defaults are useful to reduce packet sizes and processing time, they tend to encourage implementation errors.

Counter-intuitively, Postel's robustness principle ("be conservative in what you send, liberal in what you accept") often leads to deployment problems. Why? When a new implementation is initially fielded, it is likely that it will encounter only a subset of existing implementations. If those implementations follow the robustness principle, then errors in the new implementation will likely go undetected. The new implementation then sees some, but not widespread deployment. This process repeats for several new implementations. Eventually, the not-quite-correct implementations run into other implementations that are less liberal than the initial set of implementations. The reader should be able to figure out what happens next.

Accordingly, explicit consistency checks in a protocol are very useful, even if they impose implementation overhead.

On the Design of Application Protocols

I bring this up because I recently came across this:

… The first byte of the clear data was a flags field which, he said, was promptly ossified by a middlebox vendor, leading to packets being dropped when a new flag was set.

That was a classic example of why changing network protocols is hard and what needs to be done to improve the situation. Middleboxes are the control points for the Internet as we know it now. The only defense against ossification of network protocols by middleboxes, he said at the conclusion of the talk, is encryption.

Via Hacker News, QUIC as a solution to protocol ossification [LWN.net]

And I thought, Oh crap! I do that!

More specifically, in SPCDNS where I explicitly check for unused bits being zero.

Am I ossifying the DNS protocol by doing this?

I did encounter this a few months ago. I was playing around with monitoring the traffic on the home network and saw a bunch of MDNS requests flying around. I thought I would decode them using SPCDNS to see what exactly was being queried when I hit a snag—some of the packets were invalid! Upon investigation, it came down to that single undefined bit in the DNS packet being set. I quickly patched the code (privately) to accept the packets but I didn't release the changes since at the time, that last undefined bit was still undefined.

I didn't know how to deal with the situation then, and I still don't. SPCDNS can handle records it doesn't understand, but that one undefined bit has me stymied. I could define the flag and let a user of my code check for it, but when it does become defined, that flag name will obviously have to change and that could be a breaking change. But as of right now, any user code that wants to check for that bit has no way of doing so (not that I've received any requests for it).

Then there's the fact that because my code was so fussy with verifying the packet, that bugs in other DNS implementations were found via my code. And at work, I put up a fight to remain picky in processing network packets for “Project: Sippy-Cup.” I mean, what's the point of a protocol specification if you don't follow it? I even referenced RFC-3177 (quoted above) as part of my argument for being picky. But I was overridden and had to carry on in the face of utter garbage (and yes, some of what we get is utter garbage). I did however keep track of each violation and we are in the process of talking with a vendor who is sending us garbage as I type.

This is tough.

At the time I wrote SPCDNS, there was only one undefined bit left in the DNS packet. Had I written it back when RFC-1035 was written, there would have been three undefined bits, and I ask myself how I would have handled that over the years. I'm not even sure if I would have thought to include those bits in what is decoded (I don't now).

This will require some thought.

Sunday, February 11, 2018

“I'm sorry. We can't take your order right now, the computer crashed and we have no idea how to proceed.”

Today is Game Day™. Every other week I get together with some friends to play D&D for a few hours. Sometimes it's hosted here at Chez Boca, other times one of my friend's house. Today it's not at Chez Boca, so as usual, I stop by the Scottish place for a quick lunch. And because I'm weird, I avoid the drive thru.

I noticed a few months ago that McDonald's had installed a few ordering kiosks. Aside from the clunkiness of the interface, it's not that bad. Which upon reflection is strange, because I hate the things at other restaurants. My experience using the Ziosk (not only at the Olive Garden but also Chili's) is not good. A few times the unit has been broken and it took a very confused staff longer to process the check than normal.

And heaven help you if you want to pay by cash.

Cash!

Ha!

So, why do I hate the Ziosks and yet, don't mind the kiosks at McDonald's? I've been thinking about that today, and the best I can come up with is that at a sit-down restaurant, I'm there to enjoy a meal with company and don't want to be distracted by a device not only demanding our attention, but taking up space on a table that can get quite crowded with plates and glasses. I think it also cheapens the waitrons in a way—no longer do you have a single person handling your dinner, but you have several people who cycle through, one person taking the order, one person serving the drinks, another bringing the food, there's not continuity in service. And there's this feeling of replaceability of not only the servers, but of the customers.

And who is the tip for, exactly?

It's hard to describe, and maybe it's just me.

There's also the issue of letting a device, which processes financial transactions into the hands of the customers! Customers which may attempt to hack the devices (although why do I trust the waitrons?). And if the Ziosk crashes … well … I've experienced the chaos it can cause with the staff.

But the kiosk at McDonald's though—there's no tipping. The staff you interact with is just there to take your order. There are other people who do the actual job of preparing your order. And even they don't prepare the entire order. In order to save time we customers are now mostly responsible for filling our own cups of whatever lovely beverage we chose to drink. So that's one job less they have to do. And the kiosks now remove one to two other jobs—taking the order and receiving payment (if you want to pay cash, you have the option to go to the register and pay). So pretty much all that's now left is food preparation.

And I'm fine with that for fast food. Maybe because it's somewhat impersonal to begin with.

All this came to a head today as I was grabing lunch on the way to the D&D game. All the kiosks at The Scottish place had crashed. And there was much confusion at the counter.

Ah, computer automation! Gotta love it.

Monday, January 22, 2018

“It's a movie about food poisoning on an airplane, but that's not important right now.”

Airplane” is one of the funniest movies ever made and it still holds up over 35 years later. Most people think it's a parody of all the airline disaster movies of the 70s, but in reality, it's a remake of a 1957 dramatic movie played straight (which explains why the main character is a World War II fighter pilot in a movie made in 1980).

Wednesday, January 10, 2018

You know, the ability to comment on-the-fly configuration changes in long running daemons is probably a good idea

And speaking of the greylist daemon, I can filter by IP address:

[spc]brevard:~>gld-mcp show iplist
     30905 GREYLIST         0.0.0.0         0.0.0.0
       115   REJECT  61.175.186.125 255.255.255.255
         0   ACCEPT       64.12.0.0     255.255.0.0
         0   ACCEPT    64.233.160.0   255.255.224.0
         0   ACCEPT     66.94.224.0   255.255.224.0
         0   ACCEPT      66.102.0.0   255.255.240.0
        33   ACCEPT    66.163.160.0   255.255.224.0
         0   ACCEPT     66.218.64.0   255.255.224.0
        81   ACCEPT  66.220.144.128 255.255.255.128
         0   ACCEPT     66.249.80.0   255.255.240.0
         0   ACCEPT     66.249.64.0   255.255.224.0
         0   ACCEPT    66.252.224.0   255.255.252.0
         0   ACCEPT     69.63.184.0 255.255.255.128
         0   ACCEPT     69.63.176.0   255.255.240.0
         0   ACCEPT     69.147.64.0   255.255.192.0
         4   ACCEPT      70.34.16.0   255.255.240.0
         0   ACCEPT     72.14.192.0   255.255.192.0
        16   ACCEPT      74.125.0.0     255.255.0.0
         0   ACCEPT       127.0.0.1 255.255.255.255
      1207   ACCEPT    140.211.11.3 255.255.255.255
         0   ACCEPT     149.174.0.0     255.255.0.0
        12   REJECT     172.128.0.0     255.128.0.0
         7 GREYLIST     173.232.0.0     255.255.0.0
       115 GREYLIST       173.0.0.0       255.0.0.0
         0   ACCEPT     192.168.0.0     255.255.0.0
         0   ACCEPT   204.127.217.0   255.255.255.0
         0   ACCEPT     204.127.0.0     255.255.0.0
         0   ACCEPT    205.152.58.0   255.255.254.0
         0   ACCEPT   205.188.156.0   255.255.254.0
         0   ACCEPT     205.188.0.0     255.255.0.0
         0   REJECT    206.214.64.0   255.255.224.0
         0   ACCEPT    207.115.11.0 255.255.255.192
         0   ACCEPT     207.115.0.0   255.255.192.0
         0   ACCEPT   207.171.188.0   255.255.255.0
       116   ACCEPT    209.85.128.0   255.255.128.0
         0   ACCEPT    209.131.32.0   255.255.224.0
         3   ACCEPT     216.39.48.0   255.255.240.0
         0   ACCEPT    216.239.32.0   255.255.224.0
[spc]brevard:~>

But there's a question I can't answer—why? Why did I accept 69.63.184.0/19? Or 216.39.48.0/20? Heck, I can't even answer when!

What I wish I had done back when I wrote this was allow comments for this filter table. I mean, it's pretty obvious with some of the other filters, such as the to-domain list:

[spc]brevard:~>gld-mcp show to-domain
         0   REJECT area51.conman.org
        75   REJECT linus.area51.conman.org
     30530 GREYLIST DEFAULT
[spc]brevard:~>

The machine linus.area51.conman.org has been decommissioned for over a decade, and yet some spammers somewhere are still attempting to deliver email to it (although it may help if I were to remove the MX and A DNS records for it—hmm …) even after all these years (75 attempts have been attempted since September 6th, 2017 when for whatever reason, the greylist daemon was restarted). It's obvious why I added the line.

But the IP addresses? No idea. The ability to add comments for ad-hoc, on-the-fly updates for long running daemons is something to keep in mind. I just wish I had thought of that sooner.


I still find SPF not worth integrating into my email server

The other day, I came across this article about using SPF to avoid large delays using greylisting (link via Lobsters). I've experienced those delays from time to time since I run a greylist daemon and this does seem like a neat idea. But I've already ironed out those large delays (it's been awhile since the last one) so I have very little incentive to implement this.

This did however get me to ask if I should integrate SPF with the greylist daemon.

Then I thought, didn't I already do this?

It turns out I did! Three years ago.

Well … okay then!

Have things changed over the past three years? I spent a few hours over the past few days getting the code up and running, so let's see—over the past month (December 10th to January 10th):

Unique emails processed by the greylist daemon
Emails accepted2,515
Emails rejected2,319
Total 4,834

Aside from the volume (which appears to be a bit less than 50% of what it was three years ago, but that could be due to the time of year), the ratio is similar—50% of the email is spam that is immedately filtered out.

The one major change though, is that back then, around 12½% only had SPF records, whereas today, it's about 12½% don't have SPF records. That's interesting. But the rest of the figures are again, similar to last time:

Results of applying SPF policy against incoming email
fail93IP address was not allowed to send this email
softfail79IP address should not be sending this email (used for testing)
neutral11IP address has no policy
pass962IP address is allowed to send this email

Those with a neutral SPF policy have gone down, and the failure rate is up a bit (4% vs. 1% back then) but the overall conclusion is the same—not worth it.

Tuesday, January 09, 2018

So tonight I'm gonna Internet like it's 1993

A few days ago I recived this email (which was not directly addressed to me):

From
Rouan Lucas van Ryn <XXXXXXXXXXXXXXXX>
To
XXXXXXXXXXXXXXX
Subject
The Gopherspace Cookbook
Date
Sat, 6 Jan 2018 08:47:45 +0000

Greetings good people,

I have recently created a new Gopher hole, known as The Gopherspace Cookbook. It is a compendium of recipes from around Gopherspace. You're receiving this message to let you know that one or more of your recipes has been listed in the cookbook.

The cookbook is available at:
gopher://tellus.strangled.net/1/cookbook

Keep on cooking!
Keep on Gophering!

Feel free to get in touch if you have any queries.

Best,
Rouan aka jandal

--
gopher://grex.org/1/%7ejandal
gopher://tellus.strangled.net/

It's not spam, as the post in question, does have a recipe (if you are having problems with the link, don't worry, that is expected, but in the meantime, you will probably have more luck with this link). The links in question deal with Gopher, a protocol invented at the same time as HTTP, but was soon eclipsed as you couldn't inline cat pictures in a gopher page unlike a webpage. It's mostly dead.

But not all dead.

I'm surprised [No he's not. —Editor] that I haven't mentioned this before [He was being lazy. —Editor], but I set up a server to mainly serve up my blog via gopher: early last March. And if pressed for a reason—it was because I could. I thought it might be fun to have a phlog in addition to a blog.

It's not like it was hard to write the code (alternate link). It just took a day or so for me to get it working, and most of the conversion from HTML to plain text is done by Lynx (which is why it looks the way it looks; alternate link). And I do get the occasional hits on the gopher server, which I find amusing.

And yes, people do still use Gopher.

Obligatory Picture

[Here I am, enjoying my vacaton in a rain forest.]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2018 by Sean Conner. All Rights Reserved.