Problem with parallel make

[Lee Killough]

I have a makefile similar to this:

##############################################################################

J = j1 j2 j3 j4

all: $J

$J:
        +exit `(($(MAKE) address@hidden 2>&1 3>&-; echo $$? >&3 3>&-) | tee -a 
logfile) 3>&1 >&2`

$(addsuffix _,$J):
        @echo "Building $@"
        @sleep 1
        @echo "Done building $@"

.PHONY: all $J $(addsuffix _,$J)

##############################################################################

When I run parallel make, I always get the message:

make[1]: warning: jobserver unavailable: using -j1.  Add `+' to parent make 
rule.

I can find no documentation on the + option, other than the documentation
on the message, which says to use +.

The problem appears to be caused by the complicated command I have, which
concatenates the output of the recursive make into a logfile while also
printing it on the terminal.

I read somewhere that the presence of the string $(MAKE) in the unexpanded
command was all that was necessary for make to detect recursive commands,
but that appears false.

If I replace the complicated command with a simpler one, even one with
subshells, e.g.:

        ( ( ( ( ( $(MAKE) address@hidden ) ) ) ) ) 

or:

        true `$(MAKE) address@hidden       

then the recursion still gets detected. Apparently the nesting of the
shells and the command substitution is not the problem.

As you know the exit code of a pipeline is the exit code of the last
command in the pipeline, so to preserve the exit code of the recursive
make I have to capture it somehow. I do not want to use named temporary
files, because this will be done in parallel with as many as 16 CPUs, and
then I'd have race conditions (I do not mind if the log file's contents
are slightly interleaved).

PS. I just answered my own question. Question and answer here for others'
reference, although I'd still like more detailed information about the
precise semantics of + and the jobserver's operation:

Filehandle 3 was the problem. Apparently make uses filehandle 3 to talk to
the jobserver. I was using it and then closing it. I would have thought my
operations would have remained confined to my shell commands, since I
opened and closed 3 myself, but apparently filehandle 3 was already open
before my shell command started, and must be left open for the recursive
make.

Perhaps a larger filehandle number, less likely to occur in common shell
redirection idioms, should be used for the jobserver. No matter how
complicated the redirection sequence 4>&3 3>&2 2>&1 etc., at least one
filehandle must be destroyed. Others can be duped and restored, but one
must be permanently affected. Maybe sockets should be used, and the port
number passed via MAKEFLAGS.

I have printed the environment variables from my shell command, and see:

MAKEFLAGS= --jobserver-fds=3,4 -j

... which apparently means 3 and 4 are both used, perhaps one for reading
and one for writing (I've never looked at the make source).

Just something to watch out for.

Thanks,
Lee