The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Monday, February 19, 2018

Why one might want to load dynamic objects from memory instead of from disk

Years ago I started on a little project entitled the LEM Project. It was a project to make distribution of applications written in Lua easier (It's named after the Apollo Lunar Module—aka the Lunar Excursion Module; Lua is “moon” in Portuguese). I had a few goals:

  1. allow distribution of multiple versions of a Lua module;
  2. allow for different architectures for the same module written in C;
  3. run the entire application from the distribution file (nothing to install).

And I almost hit all the goals.

The first two were easy. I read over the ZIP file specification and saw that

  1. multiple files of the same name were entirely possible—nothing in the specification disallows it;
  2. additional metadata can be added to each file beyond what is defined in the specification.

So armed, I created zip file with some addtional metadata and …

[spc]lucy:~/projects/LEM>./zipf.lua sample.lem
The Eagle Has Landed
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.env
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.errno
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.1    MODULES/org.conman.iconv
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.sys
SunOS     sparcv9   LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.base64
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.crc
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.env
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.0  MODULES/org.conman.errno
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.fsys.magic
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.hash
Linux     x86       LGPL3+       Lua 5.1 5.1 1.1.1  MODULES/org.conman.iconv
Linux     x86       LGPL3+       Lua 5.1 5.1 5.1    MODULES/org.conman.math
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.net.ipacl
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.pollset
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.process
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.strcore
Linux     x86       LGPL3+       Lua 5.1 5.1 1.2.0  MODULES/org.conman.sys
Linux     x86       LGPL3+       Lua 5.1 5.1 1.0.2  MODULES/org.conman.syslog
Linux     x86       LGPL3+       Lua 5.1 5.1        MODULES/org.conman.tcc
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.cc
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.date
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.debug
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.dns.resolv
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.getopt
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.string
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.table
                    LGPL3+       Lua 5.1 5.1        MODULES/org.conman.unix
                    MIT          Lua 5.1 5.1 0.10   MODULES/lpeg
                    MIT          Lua 5.1 5.1 0.10   MODULES/re
Linux     x86       MIT          Lua 5.1 5.1 0.12   MODULES/lpeg
Linux     x86       MIT          Lua 5.2 5.2 0.12   MODULES/lpeg
                    MIT          Lua 5.1 5.2 0.12   MODULES/re
Linux     x86       MIT/X11      Lua 5.1 5.1 0.4.work3 MODULES/zlib
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket
Linux     x86       MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.core
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.ftp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.http
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.smtp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.tp
                    MIT/X11      Lua 5.1 5.1 2.0.2  MODULES/socket.url
                    MIT/X11      Lua 5.1 5.1 1.0.1  MODULES/ltn12
                                                    FILES/APPNOTE.TXT
                                                    FILES/COPYING
                                                    FILES/README
                                                    FILES/Miscellaneous Things About Nothing
[spc]lucy:~/projects/LEM>

You can see there are several copies of each module. The modules listed without a system (like Linux or SunOS) are written in Lua; the other ones are in C and both the system and architecture are listed. The license is included, along with the Lua verions the module will work for (it's “minimum version” and “maximum version”) as well as the version of the module (if known).

It's even a valid zip file:

[spc]lucy:~/projects/LEM>unzip -l sample.lem
Archive:  sample.lem
The Eagle Has Landed
  Length     Date   Time    Name
 --------    ----   ----    ----
    25472  05-24-14 17:10   MODULES/org.conman.base64
    13448  05-24-14 17:10   MODULES/org.conman.crc
    12200  05-24-14 17:10   MODULES/org.conman.env
    18688  05-24-14 17:10   MODULES/org.conman.errno
    57032  05-24-14 17:10   MODULES/org.conman.fsys
    24952  05-24-14 17:10   MODULES/org.conman.hash
    17664  05-24-14 17:10   MODULES/org.conman.iconv
    17648  05-24-14 17:10   MODULES/org.conman.math
    77944  05-24-14 17:10   MODULES/org.conman.net
    26296  05-24-14 17:10   MODULES/org.conman.pollset
    88256  05-24-14 17:10   MODULES/org.conman.process
    37848  05-24-14 17:10   MODULES/org.conman.strcore
    15392  05-24-14 17:10   MODULES/org.conman.sys
    24312  05-24-14 17:10   MODULES/org.conman.syslog
    14175  06-12-14 22:04   MODULES/org.conman.base64
     8214  06-12-14 22:04   MODULES/org.conman.crc
     7193  06-12-14 22:04   MODULES/org.conman.env
    10690  06-12-14 22:04   MODULES/org.conman.errno
    31885  06-12-14 22:04   MODULES/org.conman.fsys
    15567  06-12-14 22:04   MODULES/org.conman.fsys.magic
    14067  06-12-14 22:04   MODULES/org.conman.hash
    10197  06-12-14 22:04   MODULES/org.conman.iconv
    10816  06-12-14 22:04   MODULES/org.conman.math
    43651  06-12-14 22:04   MODULES/org.conman.net
    25248  04-18-14 20:06   MODULES/org.conman.net.ipacl
    15914  06-12-14 22:04   MODULES/org.conman.pollset
    49607  06-12-14 22:04   MODULES/org.conman.process
    15666  06-12-14 22:04   MODULES/org.conman.strcore
     9763  06-12-14 22:04   MODULES/org.conman.sys
    13303  06-12-14 22:04   MODULES/org.conman.syslog
    21218  06-12-14 22:04   MODULES/org.conman.tcc
     5617  06-12-14 22:04   MODULES/org.conman.cc
     2500  06-12-14 22:04   MODULES/org.conman.date
     3829  06-12-14 22:04   MODULES/org.conman.debug
     2966  06-12-14 22:04   MODULES/org.conman.dns.resolv
     3260  06-12-14 22:04   MODULES/org.conman.getopt
     2464  06-12-14 22:04   MODULES/org.conman.string
     5243  06-12-14 22:04   MODULES/org.conman.table
     2732  06-12-14 22:04   MODULES/org.conman.unix
    40081  05-25-14 15:17   MODULES/lpeg
     6029  05-24-14 00:36   MODULES/re
    40045  05-28-14 15:24   MODULES/lpeg
    40045  05-28-14 15:24   MODULES/lpeg
     6286  05-28-14 15:24   MODULES/re
    19794  05-30-14 21:29   MODULES/zlib
     4451  05-28-14 14:52   MODULES/socket
    55449  05-28-14 14:52   MODULES/socket.core
     9243  05-28-14 14:52   MODULES/socket.ftp
    12330  05-28-14 14:52   MODULES/socket.http
     8074  05-28-14 14:52   MODULES/socket.smtp
     3612  05-28-14 14:52   MODULES/socket.tp
    11036  05-28-14 14:52   MODULES/socket.url
     8331  05-28-14 14:52   MODULES/ltn12
   161412  05-09-14 01:24   FILES/APPNOTE.TXT
The ZIP file format
     7651  05-25-14 18:53   FILES/COPYING
     9789  06-07-14 22:13   FILES/README
Much Ado About Nothing
     3946  05-10-99 23:00   FILES/Miscellaneous Things About Nothing
If you this this has significance, think otherwise
 --------                   -------
  1250541                   57 files
[spc]lucy:~/projects/LEM>

(Although if you try to unzip this file, the unzip program will ask you what you want to do with the duplicate files.)

The intent was that only those modules that match the Lua version, system (if a C-based Lua module) and architecture (again, C-base Lua module) match, load the module; otherwise, it would be ignored.

And it worked. For the most part.

I could load Lua modules written in Lua directly from the zip file with no problem. I even got LuaRocks to run completely from the zip file (with some hacks so it could load the configuration file). But I could not do that for the C-based Lua modules. For those, you had to write them out to disk to load them up. And because of that, I lost interest in continuing the project.

Until now. I just came across MojoELF (via Project: 2ine which was linked via Hacker News)—a project to load ELF binaries directly from memory (and those C-based Lua modules I have are ELF binaries), which is exactly what I wanted all those years ago.

Obligatory Picture

An abstract representation of where you're coming from]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

Obligatory AI Disclaimer

No AI was used in the making of this site, unless otherwise noted.

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.