The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Monday, February 19, 2018

Fun with banging moon rocks

Back in Debtember, I was notified of a potential problem with one of my Lua modules, speficically, the LuaRocks specfile of said module—it failed to compile properly on Mac OS-X.

I finally got around to it and fixed the issue, and while I was doing that, I figured I might as well fix some of the other Lua modules I have.

You can always tell when I do this because the version numbers accumulate in a rapid succession. My CBOR Lua module went from version 1.2.4 to 1.2.11 in the span of an hour or so. And it's all due to the picky little details.

As I mentioned a few months ago, I pull the version directly from git. But I also have to define the version in the Makefile in case someone downloads the code instead of cloning the repository. But there have been times when I updated the GitHub repository without updating the Makefile, or forgot to properly rename the LuaRocks specfile (probably as a sanity check, the filename needs to include the version number which is also specified inside the file). To handle those cases, I added a script to git to check for such things before pushing the update to GitHub.

So it should be easy, right?

Not quite.

One issue I forgot to check in the script was to ensure the LuaRocks specfile was syntactically correct. So there goes one version.

The Lua module is written with C99. So I specified c99 as the compiler in the LuaRocks specfile. Only the default options LuaRocks uses breaks on Mac OS-X with c99 as the compiler. The why is simple—Mac OS-X uses clang, which supports different options than GCC. But LuaRocks defaults to using GCC when compiling on Mac OS-X. That's fine, because there's a GCC wrapper on Mac OS-X that translates the options for GCC to clang, but only if the compiler you use is GCC. On Mac OS-X, c99 defaults to clang so of course it fails because clang receives options for GCC.

I can use the default compiler that LuaRocks defines on Mac OS-X and it compiles fine. But not on Linux, where GCC defaults to “not C99.” I finally figured out the magical incantation to fix that:

build =
{
  platforms =
  {
    windows =
    {
      type    = "builtin",
      modules = 
      {
        ['org.conman.cbor_c'] = 
        {
          sources = { 'cbor_c.c', 'dnf.c' },
          defines = { 'VERSION="' .. source.tag .. '"' },
        },

        ['org.conman.cbor']     = 'cbor.lua',
        ['org.conman.cbor_s']   = 'cbor_s.lua',
        ['org.conman.cbormisc'] = 'cbormisc.lua',
      }
    }
  },
  
  type = "make",
  build_variables =
  {
    CC      = "$(CC)",
    CFLAGS  = "$(CFLAGS) -DNDEBUG -I$(LUA_INCDIR)",
    LDFLAGS = "$(LIBFLAG)",
    LUA     = "$(LUA)",
  },
  
  install_variables =
  {
    LIBDIR = "$(LIBDIR)",
    LUADIR = "$(LUADIR)",
    LUA    = "$(LUA)",
  },
  
  platforms =
  {
    linux   = { build_variables = { CC = "gcc -std=c99" } },
    solaris = { build_varaibles = { CC = "c99"          } },
  }
}

And it worked fine. But it burned through some additional versions as I worked out the details. And then I found an issue—if you look closely, you'll notice that I have the patforms stanza twice. That means the second stanza replaces the first stanza and thus, I lost the Microsoft Windows compiling instructions.

So yet another version goes by as I merge the two platforms stanzas into a single block.

And then I forgot to update the version number in the LuaRocks specfile. It had the right name, but the wrong internal version.

So there's another version and an update the the checking script.

By now, you might be wondering why I don't bother testing this before updating the version number. And it's simple—I can't! The LuaRocks specfile contains this bit:

package = "org.conman.cbor"
version = "1.2.11-1"

source =
{
  url = "git://github.com/spc476/CBOR.git",
  tag = version:match "[^-]+"
}

To test it, LuaRocks will download the code from GitHub using the given tag value, which is based off the version number. But if I haven't checked in the code for that tag, LuaRocks will fail. So I have to check in the code with the proper tag before I can test the LuaRocks file.

Yes, I could do the code updates and afterwards fix the LuaRocks specfile, but the script that's run prior to pushing changes checks to see if the latest version is a “good version,” like 1.2.10. It will reject a version like 1.2.10-1-g5b75b4f (which is git's way of saying “one commit past the previous tagged version”). I added that check to ensure LuaRocks would get the proper version of everything—I would only push when I had everything ready for a new version.

Ah—unintended consequences.

We'll see if the next update goes smoother. One of these days I'll get this right.

Why one might want to load dynamic objects from memory instead of from disk

Years ago I started on a little project entitled the LEM Project. It was a project to make distribution of applications written in Lua easier (It's named after the Apollo Lunar Module—aka the Lunar Excursion Module; Lua is “moon” in Portuguese). I had a few goals:

  1. allow distribution of multiple versions of a Lua module;
  2. allow for different architectures for the same module written in C;
  3. run the entire application from the distribution file (nothing to install).

And I almost hit all the goals.

The first two were easy. I read over the ZIP file specification and saw that

  1. multiple files of the same name were entirely possible—nothing in the specification disallows it;
  2. additional metadata can be added to each file beyond what is defined in the specification.

So armed, I created zip file with some addtional metadata and …

[spc]lucy:~/projects/LEM>./zipf.lua sample.lem
The Eagle Has Landed
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.env
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.errno
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.1    MODULES/org.conman.iconv
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.sys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.env
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.errno
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys.magic
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
Linux     x86       LGPL3+       Lua 5.1 5.1 1.1.1  MODULES/org.conman.iconv
Linux     x86       LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net.ipacl
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
Linux     x86       LGPL3+       Lua 5.1 5.1 1.2.0  MODULES/org.conman.sys
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.2  MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.tcc
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.cc
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.date
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.debug
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.dns.resolv
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.getopt
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.string
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.table
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.unix
                    MIT          Lua 5.1 5.1 0.10   MODULES/lpeg
                    MIT          Lua 5.1 5.1 0.10   MODULES/re
Linux     x86       MIT          Lua 5.1 5.1 0.12   MODULES/lpeg
Linux     x86       MIT          Lua 5.2 5.2 0.12   MODULES/lpeg
                    MIT          Lua 5.1 5.2 0.12   MODULES/re
Linux     x86       MIT/X11      Lua 5.1 5.1 0.4.work3 MODULES/zlib
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket
Linux     x86       MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.core
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.ftp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.http
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.smtp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.tp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.url
                    MIT/X11      Lua 5.1 5.1 1.0.1  MODULES/ltn12
                                                    FILES/APPNOTE.TXT
                                                    FILES/COPYING
                                                    FILES/README
                                                    FILES/Miscellaneous Things About Nothing
[spc]lucy:~/projects/LEM>

You can see there are several copies of each module. The modules listed without a system (like Linux or SunOS) are written in Lua; the other ones are in C and both the system and architecture are listed. The license is included, along with the Lua verions the module will work for (it's “minimum version” and “maximum version”) as well as the version of the module (if known).

It's even a valid zip file:

[spc]lucy:~/projects/LEM>unzip -l sample.lem
Archive:  sample.lem
The Eagle Has Landed
  Length     Date   Time    Name
 --------    ----   ----    ----
    25472  05-24-14 17:10   MODULES/org.conman.base64
    13448  05-24-14 17:10   MODULES/org.conman.crc
    12200  05-24-14 17:10   MODULES/org.conman.env
    18688  05-24-14 17:10   MODULES/org.conman.errno
    57032  05-24-14 17:10   MODULES/org.conman.fsys
    24952  05-24-14 17:10   MODULES/org.conman.hash
    17664  05-24-14 17:10   MODULES/org.conman.iconv
    17648  05-24-14 17:10   MODULES/org.conman.math
    77944  05-24-14 17:10   MODULES/org.conman.net
    26296  05-24-14 17:10   MODULES/org.conman.pollset
    88256  05-24-14 17:10   MODULES/org.conman.process
    37848  05-24-14 17:10   MODULES/org.conman.strcore
    15392  05-24-14 17:10   MODULES/org.conman.sys
    24312  05-24-14 17:10   MODULES/org.conman.syslog
    14175  06-12-14 22:04   MODULES/org.conman.base64
     8214  06-12-14 22:04   MODULES/org.conman.crc
     7193  06-12-14 22:04   MODULES/org.conman.env
    10690  06-12-14 22:04   MODULES/org.conman.errno
    31885  06-12-14 22:04   MODULES/org.conman.fsys
    15567  06-12-14 22:04   MODULES/org.conman.fsys.magic
    14067  06-12-14 22:04   MODULES/org.conman.hash
    10197  06-12-14 22:04   MODULES/org.conman.iconv
    10816  06-12-14 22:04   MODULES/org.conman.math
    43651  06-12-14 22:04   MODULES/org.conman.net
    25248  04-18-14 20:06   MODULES/org.conman.net.ipacl
    15914  06-12-14 22:04   MODULES/org.conman.pollset
    49607  06-12-14 22:04   MODULES/org.conman.process
    15666  06-12-14 22:04   MODULES/org.conman.strcore
     9763  06-12-14 22:04   MODULES/org.conman.sys
    13303  06-12-14 22:04   MODULES/org.conman.syslog
    21218  06-12-14 22:04   MODULES/org.conman.tcc
     5617  06-12-14 22:04   MODULES/org.conman.cc
     2500  06-12-14 22:04   MODULES/org.conman.date
     3829  06-12-14 22:04   MODULES/org.conman.debug
     2966  06-12-14 22:04   MODULES/org.conman.dns.resolv
     3260  06-12-14 22:04   MODULES/org.conman.getopt
     2464  06-12-14 22:04   MODULES/org.conman.string
     5243  06-12-14 22:04   MODULES/org.conman.table
     2732  06-12-14 22:04   MODULES/org.conman.unix
    40081  05-25-14 15:17   MODULES/lpeg
     6029  05-24-14 00:36   MODULES/re
    40045  05-28-14 15:24   MODULES/lpeg
    40045  05-28-14 15:24   MODULES/lpeg
     6286  05-28-14 15:24   MODULES/re
    19794  05-30-14 21:29   MODULES/zlib
     4451  05-28-14 14:52   MODULES/socket
    55449  05-28-14 14:52   MODULES/socket.core
     9243  05-28-14 14:52   MODULES/socket.ftp
    12330  05-28-14 14:52   MODULES/socket.http
     8074  05-28-14 14:52   MODULES/socket.smtp
     3612  05-28-14 14:52   MODULES/socket.tp
    11036  05-28-14 14:52   MODULES/socket.url
     8331  05-28-14 14:52   MODULES/ltn12
   161412  05-09-14 01:24   FILES/APPNOTE.TXT
The ZIP file format
     7651  05-25-14 18:53   FILES/COPYING
     9789  06-07-14 22:13   FILES/README
Much Ado About Nothing
     3946  05-10-99 23:00   FILES/Miscellaneous Things About Nothing
If you this this has significance, think otherwise
 --------                   -------
  1250541                   57 files
[spc]lucy:~/projects/LEM>

(Although if you try to unzip this file, the unzip program will ask you what you want to do with the duplicate files.)

The intent was that only those modules that match the Lua version, system (if a C-based Lua module) and architecture (again, C-base Lua module) match, load the module; otherwise, it would be ignored.

And it worked. For the most part.

I could load Lua modules written in Lua directly from the zip file with no problem. I even got LuaRocks to run completely from the zip file (with some hacks so it could load the configuration file). But I could not do that for the C-based Lua modules. For those, you had to write them out to disk to load them up. And because of that, I lost interest in continuing the project.

Until now. I just came across MojoELF (via Project: 2ine which was linked via Hacker News)—a project to load ELF binaries directly from memory (and those C-based Lua modules I have are ELF binaries), which is exactly what I wanted all those years ago.

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.