A build system using ``Recursive Make Considered harmful''

[Scott A Crosby]

Hello.

I read the ``Recursive Make Considered harmful'' paper[1] about a year ago.

As I was recently faced with creating my first large codebaes, I needed a
build system. I decided to do one the 'right way', so I could reuse it for
future projects. I implemented the scheme described in the paper. Here,
give my build system and my observations of it in use. Overall, it is very
pleasant to use; it is error-free in parallel-make configurations, it does
full dependency information automatically, and it always does the minimum
work necessary during building.


To satisfy GNU standards, it would need to be adapted and changed. I
cannot do this work; I do not know all the subtle issues with Make, nor am
I really interested in learning.

But I do offer this as a starting point for someone else.

I am including the full buildsystem I use. This is all of my makefiles,
but no source code.

There are two issues dealing with subtleties of make semantics that I
would appreciate help and/or suggestions in fixing. They are with
program-generated code and with fragility during cleaning.

If anyone has any suggestions for fixes to these, I'll be happy to test
them and post up another tarball later.

Finally, please keep me on the CC in these messages.

-- What I have

I currently have about 30 .cpp and 40 .h files in 5 directories, and I've
followed the advice given. I build 10 binaries and 2 libraries. If there
is nothing to be done, it takes make .04 seconds to realize this.  This
includes 140 stat's, and opening 50 files for including. Admittiedly this
isn't the largest project thats been done, but it is respectible in size.

Dependencies are built fully automatically, in parallel and take
2.5 seconds.

The toplevel makefile, which deals with 4 types of C++ source code (the
different types use different options during compilation) is 168 lines.
The subdirectory makefiles are 1-40 lines. Total size of all makefiles is
257 lines and 7kb.

As per the paper, make only processes the minimum amount is has to. I
also have no worries or problems with parallel building.

Finally, the makefile system is pretty simple. I do not deal with making
sure that subdirectory makes get the right flags and options or get built
in the right order. Nor, with one exception, do I have any shell scripts
in the makefile. (I do have one ugly shell script responsible for getting
and fixing-up dependency information.)

Overall, its very sweet.

I am using
  GNU Make version 3.79.1, by Richard Stallman and Roland McGrath.
  Built for i586-pc-linux-gnu

As I am doing this under linux, I have not been concerned about makefile
portability to other systems. Thus, I may have used GNU extensions without
intending so.

-- Problems or untested issues

Overall, I've been extremely happy with it, though it is flawed in a
couple of ways:

First, 'make clean' builds dependencies before doing any cleaning. This is
both slow, and fragile in the case where you rename a file, and make
attempts to build dependencies on a file that no longer exists. Make
breaks. One workaround is '-k', or run 'make clean cleandep' before
renaming or deleting files. A suggested fix for this would be appreciated!

Second, All subdirectory modules must be declared in the toplevel
makefile, even recursive modules. No subdirectory makefile can add new
directories into the search path. This is probably not to severe,
something like autoconf can build the full list for me.

Third, I cannot use '%' in any filename or path name. (see below for the
reason why)

Currently, I am not seperating out the build directory from the run
directory, though that could be done fairly easily in the toplevel build
rules. My design is that make keeps its current directory always be the
toplevel, so it uses full paths to all subfiles.

I think the build system could be scaled to handle much larger
professional jobs. For example, the case where people checking out a
subset of a system and use symbolic links for *.o files. to avoid having
to rebuild them unnecessarily. This could be done by limiting the
directories checked at the toplevel and/or modifying my %.o :  %.cpp rule
to remove the destination (symbolic link or not)

I have not tested it with program-built code. IE. the case where one
program is run to generate code that is later compiled.  (IE:
'./genCode interp.spec > interp.c') My problem is is with how and when do
I create and include the dependency file on the generated code.
('interp.c') Suggestions/fixes welcome.

I cannot build only a single subdirectory/submodule. as-is. This can be
emulated easily by having the subdirectory 'Foo/Makefile.dir' define a
target 'subdir_Foo' that depends on the the targets that that subdirectory
builds.

I have only not tested shared-library generation with libtool. I do not
expect this to be an issue: just alter the compilation rules for *.lo.

My project has only recently (last week or so) scaled to the point where I
wanted static libraries to help me with building. But those seem to work.

-- Bugs

I've also found a few bugs given in the paper and make texinfo:

First, dependency generation, the code given in the paper does not work if
there are '/''s in filenames. This has to be fixed to work practically.

$(patsubst %.c,%.d,$(filter %.c,$(SRC))) : %.d: %.c
        @echo "***** Doing dependencies for $<"
#       @echo "$(CC) -MM $(CPPFLAGS) $<"
        @set -e; b=`basename $* .c` ; d=`dirname $*` ; \
                $(CC) -MM $(CPPFLAGS) $< \
                | sed "s%\\($$b\\)\\.o[ :]*%$${d}/\\1.o $${d}/\\1.d : %g" > $@; 
\
                [ -s $@ ] || rm -f $@
The origional bugs were:
  1. The command-line sed pattern used a '/' as a seperator. Not good in
the case of filenames (now paths) that may include a '/'. I've changed
this to a '%', which means that '%' CANNOT be used in file names or
directory names. Please suggest other characters.
  2. 'gcc -MM' was futzing up filenames/pathnames in its output, if you
gave it something with a path 'foo/bar.cpp' the automatically generated
dependency file would make dependencies for 'bar.cpp' and lose the
directory prefix: 'foo/'. This should be reported to GCC. I have not done
this. Regardless, to deal with previous versions there would need to be an
autoconf check to see if it was broken.

Or at least this is what I think is going on... I did this 4 months ago and
have no idea how it works either. :)

The bug I refer to:

address@hidden:/tmp$ mkdir XXYYZZ
address@hidden:/tmp$ touch XXYYZZ/bar.cpp
address@hidden:/tmp$ gcc -MM XXYYZZ/bar.cpp
bar.o: XXYYZZ/bar.cpp

The paper[1] and the texinfo page for Make should be modified to include
these corrections. Also, is this a bug in GCC that should be reported?


-- The future

But, if we fix these warts, maybe we can convince people to trash the more
crazy build systems that do use recursive make?

Its nice for things to 'just work', and in complete parallelization.

So, what do you think?

Scott


[1] http://www.pcug.org.au/~millerp/rmch/recu-make-cons-harm.html

Build.tgz
Description: GNU Zip compressed data