The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, October 21, 2009

One Moon Daemon, coming up

My latest adventure into Lua has been to embed the langauge into a C application (an outgrowth of extending in C). I had this “proof-of-concept” network daemon I wrote that was simple enough to gut completely and replace the entire server logic with Lua, leaving the C code to handle the sockets.

But that wasn't hard enough. No. I wanted each connection handled by a Lua thread, and the ability to write “normal looking code” such as:

-- A simple echo server in Lua

function main(connection)
  while true do
    connection:write(connection:read("*a"))
  end
end

I will say this, unlike Theo Schlossnagle, who doesn't care for Lua the langauge but likes embedding it, I came to the opposite conclusion: I like Lua the language but embedding it is … interesting. Perhaps because I dove headfirst into Lua coroutines to get this working.

The first oddness with embedding Lua is that you push parameters onto the Lua stack, but not all lua functions pop all parameters. Namely, anything using a table won't actually pop the table parameter off the stack, and the stack notation used in the manual is not the notation I'm used to (having come to stack notations in Forth). So I'm sure that a majority of the issues I had were due to improper stack manipulations on my part.

And about that stack. You can reference the stack in two ways, from the bottom using positive indices, or from the top using negative indices, with 0 as an invalid stack index.

Lua execution stack of five elements
Bottom of Stack   Top of Stack
1
-5
2
-4
3
-3
4
-2
5
-1

As a longtime C (and assembly) programmer, it just looks weird to use 1-based indices in C.

Another horrible issue—via the C API you can push integers onto the Lua stack, you can push strings onto the Lua stack, you can push functions (either written in Lua or C) onto the stack, heck, you can push everything but userdata structures onto the Lua stack. Userdata structures are allocated by Lua (via lua_newuserdata()), can contain anything you want, but there's no way to actually push such a structure onto the Lua stack, so you always have to keep it around on the Lua stack, or in a Lua variable or table, which is a bit of a pain. I suppose it has to be that way in order for Lua to keep track of it for garbage collection purposes, but still, a pain.

And then there's the coroutines

Easy enough to create a coroutine:

lua_State *new;
Foo       *userdata;
int        rc;

/*----------------------------
; create a new thread 
; main(userdata)
;----------------------------*/

new      = lua_newthread(g_L);
lua_getglobal(new,"main");
userdata = lua_newuserdata(new,sizeof(Foo));
	/* fill in our userdata */
rc = lua_resume(new,1);

But once the thread finishes, then what?

lua_close

void lua_close (lua_State *L);

Destroys all objects in the given Lua state (calling the corresponding garbage-collection metamethods, if any) and frees all dynamic memory used by this state. On several platforms, you may not need to call this function, because all resources are naturally released when the host program ends. On the other hand, long-running programs, such as a daemon or a web server, might need to release states as soon as they are not needed, to avoid growing too large.

Lua 5.1 Reference Manual

Okay, so I'm concerned about garbage collection—I don't want tons of garbage piling up and then … the … … big … … … pause … … … … while … … … … … garbage … … … … … … collection takes place. And seeing how lua_close() mentions daemons, I figure I can call lua_close when a thread terminates, since I'm writing a daemon. But when I do, every open connection suddenly closes and any new connection causes the daemon to crash.

Huh?

lua_newthread

lua_State *lua_newthread (lua_State *L);

Creates a new thread, pushes it on the stack, and returns a pointer to a lua_State that represents this new thread. The new state returned by this function shares with the original state all global objects (such as tables), but has an independent execution stack.

There is no explicit function to close or to destroy a thread. Threads are subject to garbage collection, like any Lua object.

Lua 5.1 Reference Manual

Reading between the lines, you don't really get a new state, you get a copy of the existing state, but a new execution stack. Calling lua_close() on this “new” state is just like calling lua_close() on the already existing state.

Nope. What I wanted to do after each thread was to call lua_gc(), only now current connections are getting garbage-collected along with the finished ones, which meant I wasn't using luaL_ref() and luaL_unref() properly, probably because I wasn't managing the Lua stack properly …

Sigh.

It also took a bit of careful coding to get the Lua code to properly block (lua_yield()) and resume (lua_resume()) when you have input and output asynchronously appearing. It was also a bit tricky to have one Lua thread write to another Lua thread's socket without blowing things up. But eventually, not only did I get it such that the echo service above works (as written above) but the following simple chat script as well:

if members == nil then
  members = {}
end

local function login(socket)
  socket:write("Handle you go by: ")
  return socket:read()
end

local function wallaction(socket,who,everybody,me)
  for connection in pairs(members) do
    if connection ~= socket then
      connection:write(string.format("%s %s\n",who,everybody))
    else
      if me ~= nil then connection:write(string.format("%s\n",me)) end
    end
  end
end

local function wall(socket,who,everybody,me)
  return wallaction(socket,who .. ":",everybody,me)
end

function main(socket)
  local name      = login(socket)
  members[socket] = name
  
  wallaction(socket,name,"is in da room!","You are in the room.")
  io.stdout:write(string.format("%s has joined the party!\n",name))
  
  while true do
    wall(socket,name,socket:read())
  end
end

function fini(socket)
  io.stdout:write(string.format("%s has left the party!\n",members[socket]))
  wallaction(socket,members[socket],"has left the building!")
  members[socket] = nil
end

Each Lua thread starts with main(). fini() is called when a connection is dropped. I also made the daemon such that sending it a SIGUSR1 will cause it to re-compile the script, so that changes to the service can be made without having to restart the program as a whole (which explains the odd way I define members—if we reload the script, I don't want to lose the current members).

As written, the code isn't multi-threaded—at the C level, it's just one process round-robinning a bunch of Lua threads, so I can get away with the chat process above without worry as only one Lua thread is executing at any one time.

But it's now quite easy to write network daemons, so easy that I'll leave you with one more—an RFC compliant Quote of the Day server:

QUOTESFILE = "/home/spc/quotes/quotes.txt"
quotes     = {}

do
  local eoln = "\r\n"
  local f    = io.open(QUOTESFILE,"r")
  local s    = ""

  for line in f:lines() do
    if line == "" then
      -- each quote is separated by a blank link
      if #s < 512 then
        table.insert(quotes,s)
      end
      s = ""
    else
      s = s .. line .. eoln
    end
  end

  f:close()
end

math.randomseek(os.time())

function main(socket)
  socket:write(quotes[math.random(#quotes)])
end

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2019 by Sean Conner. All Rights Reserved.