The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Saturday, April 04, 2015

Some thoughts on make

From time to time, I find make limiting, think there has to be a better way and I start playing around with the idea of building a new make. It's not that uncommon for programmers to think this (SCons, CMake, ant, rake and a bunch of other programs) but oddly enough, as I try them, I keep going back to make (specifically, GNU make; note that when I mention make, I am talking about GNU make).

I think the main reason I go back is that make is mainly a syntax-light declarative langauge with bits of shell scripting thrown in. The other build systems are generally based around another language I do not know, they tend towards being very imperative, and rarely can you parallelize building the software (and when you get the makefile right, a parallel run of make just flies).

So when I had that “let's remake make” itch this time, I thought that perhaps I would see if I could stay within the confines of the existing syntax of make as much as possible. I wasn't trying to actually program my own make, but rather, just play around with how I would like make to look and work.

The main problem with make is declaring the dependencies. For instance, I'm embedding a UUID library into a larger project, and because recursive make is considered harmful, I include in my makefile:

lib/libspcuuid.a : build/third_party/uuid/luauuid.o	\
		build/third_party/uuid/uuid_ns_dns.o	\
		build/third_party/uuid/uuid_ns_null.o	\
		build/third_party/uuid/uuid_ns_oid.o	\
		build/third_party/uuid/uuid_ns_url.o	\
		build/third_party/uuid/uuid_ns_x500.o	\
		build/third_party/uuid/uuidlib_cmp.o	\
		build/third_party/uuid/uuidlib_parse.o	\
		build/third_party/uuid/uuidlib_toa.o	\
		build/third_party/uuid/uuidlib_v1.o	\
		build/third_party/uuid/uuidlib_v2.o	\
		build/third_party/uuid/uuidlib_v3.o	\
		build/third_party/uuid/uuidlib_v4.o	\
		build/third_party/uuid/uuidlib_v5.o

build/third_party/uuid/luauuid.o	: third_party/uuid/src/luauuid.c
build/third_party/uuid/uuid_ns_dns.o	: third_party/uuid/src/uuid_ns_dns.c
build/third_party/uuid/uuid_ns_null.o	: third_party/uuid/src/uuid_ns_null.c
build/third_party/uuid/uuid_ns_oid.o	: third_party/uuid/src/uuid_ns_oid.c
build/third_party/uuid/uuid_ns_url.o	: third_party/uuid/src/uuid_ns_url.c
build/third_party/uuid/uuid_ns_x500.o	: third_party/uuid/src/uuid_ns_x500.c
build/third_party/uuid/uuidlib_cmp.o	: third_party/uuid/src/uuidlib_cmp.c
build/third_party/uuid/uuidlib_parse.o	: third_party/uuid/src/uuidlib_parse.c
build/third_party/uuid/uuidlib_toa.o	: third_party/uuid/src/uuidlib_toa.c
build/third_party/uuid/uuidlib_v1.o	: third_party/uuid/src/uuidlib_v1.c
build/third_party/uuid/uuidlib_v2.o	: third_party/uuid/src/uuidlib_v2.c
build/third_party/uuid/uuidlib_v3.o	: third_party/uuid/src/uuidlib_v3.c
build/third_party/uuid/uuidlib_v4.o	: third_party/uuid/src/uuidlib_v4.c
build/third_party/uuid/uuidlib_v5.o	: third_party/uuid/src/uuidlib_v5.c

Or, if I wanted to save myself some typing:

define OBJECT_template = 
  $(1) : $(2)
endef

UUIDSRC = $(wildcard third_party/uuid/src/*.c)

lib/libspcuuid.a : $(subst third_party/uuid/src,build/third_party/uuid,$(UUICSRC))

$(foreach target,$(UUIDSRC),$(eval $(call OBJECT_template(build/third_party/uuid/$(notdir $(target)),$(target)))))

and have no idea what is going on three months from now (if indeed, I got that right). Instead, I would like to type:

lib/libspcuuid.a : build/third_party/uuid/*.o
build/third_party/uuid/*.o : third_party/uuid/src/*.c

and be done. This should be possible. “lib/libspcuuid.a” doesn't exist, but it depends upon all the “.o” files in “build/third_party/uuid”, which don't exist, but we have a rule that says “for ‘.o” files in ‘build/third_party/uuid’, there should be a corresponding ‘.c” file in ‘third_party/uuid/src/’ that is is generated from.” Which also leads to another problem with make: if “build/third_party/uuid” doesn't exist, make should make it! Automatically! I mean, make has no problems with “uuidlib_v5.o” not existing—that's the reason we're using make in the first place, to make files! If a directory a file lives in doesn't exist, make should make that too. Right?

By the same token, if I wanted to include some Lua modules but not all of them, I would like to do:

SPCMODC = env errno fsys hash math net pollset process strcore sys syslog
SPCMODL = debug getopt string table unix

lib/libspcmod.a : build/third_party/lua-conmanorg/*.o

build/third_party/lua-conmanorg/*.o : third_party/lua-conmanorg/src/{ $(SPCMODC) }.c \
	third_party/lua-conmanorg/lua/{ $(SPCMODL) }.lua

(Ah! So that's where yesterday's code snippet came from!)

I'll spare you the expansion.

Another small bit of annoyance is writing implicit rules to build the files. I learned the hard way that in make, this:

%.o : %.c
	$(CC) $(CFLAGS) -c -o $@ $<

only works if the “.c” and “.o” files are in the same directory! So the above rules works fine for:

long/path/to/src/foo.o : long/path/to/src/foo.c
src/bar.o              : src/bar.c
snafu.o                : snafu.c

But not for:

build/foo.o    : long/path/to/src/foo.c
build/bar.o    : src/bar.c
build/snafnu.o : snafu.c

Nope, I also have to write:

build/%.o : long/path/to/src/*.c
	$(CC) $(CFLAGS) -c -o $@ $<

build/%.o : src/%.c
	$(CC) $(CFLAGS) -c -o $@ $<

build/%.o : %.c
	$(CC) $(CFLAGS) -c -o $@ $<

if I want to segregate the “.o” files from the “.c” files.

How hard is it to realize that if I have a “.c” file, then the command to make a “.o” file is the same, regardless of where the “.c” file comes from, or where the “.o” file is being generated. Hello?

And while we're on that subject, one implicit rule I use is the following:

%.o : %.lua
	$(LUAC) -o $(@D)/$(*F).out $<
	$(BIN2C) -9 -o $(@D)/$(*F).c -t $(*F) $(@D)/$(*F).out
	$(CC) $(CFLAGS) -c -o $@ $(@D)/$(*F).c
	$(RM) $(@D)/$(*F).out $(@D)/$(*F).c

That bit of jibberish first compiles the Lua code using luac, then that output is converted to a C file which is then compiled into a object file so I can link it into the final executable, them removes the temporary files it needed to do all that. The noise you see is the cruft necessary to specify temporary files that won't get overwritten when doing a parallel build. Much nicer would be something like:

%.o : %.lua
	$(LUAC) -o $$.1 $<
	$(BIN2C) -9 -o $$.2 -t $(*F) $$.1
	$(CC) $(CFLAGS) -c -o $@ $$.2
	$(RM) $$.1 $$.2

(even better—make automatically deletes the intermediate files unless I tell it not to)

One complaint about make (or rather, makefiles themselves) is that a change to the makefile does not cause a recompile, mainly because the makefile itself is never listed as a dependency anywhere. This is never what you want! If the makefile was listed as a depenency (either explicitly or implicitly), then adding a new file to be compiled in would cause everything to be recompiled, when it should be the new file, any files that call code in the new file, and a final relinking of the code is all that is needed. Also, if you change a variable, like $(LDFLAGS), then all that should happen is any rule that uses $(LDFLAGS) should be run, not a complete build.

In other words, you really have a dependency on a portion of the makefile, not the makefile as a whole. But solving this would require a possible rethink of how make works (perhaps a cached version of the makefile that mode checks against, and intelligently applies the changes as dependencies? You know, so if you change $(CC), any rule using $(CC) is automatically run).

I realize these some of my ideas are not that easy to implement (heck, I wasted a few hours yesterday writing code to properly build a list of targets and dependencies based on ideas presented here), but I think they may go a long way to making make less horrendous to use.

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2021 by Sean Conner. All Rights Reserved.