Why not solve the vpath missing directory problem?

[Bryan Ischo]

Hey all.  I ran into a problem with my Makefile recently and I searched
the archives of this list to find that several other people have also had
the same problem.  The problem can be demonstrated by this small Makefile:

--- begin Makefile ---

.PHONY: notarget
notarget:
        $(error no target specified)

vpath %.c src-generated

src-generated/main-generated.c: src/main.c
        mkdir -p src-generated
        cp $< $@

test: main-generated.c
        gcc -o $@ $^

.PHONY: clean
clean:
        rm -rf src-generated test

--- end Makefile ---

This Makefile fragment illustrates a typical manner in which generated
source code might be built using make.

If I start with a directory structure that looks like this:

$ tree
.
|-- Makefile
`-- src
    `-- main.c

1 directory, 2 files

Then here's the output of make with this Makefile:

$ make test
make: *** No rule to make target `main-generated.c', needed by `test'.  Stop.

The problem here is that we're relying on vpath to find make-generated.c
for us.  You'd think that the vpath for .c files would locate
main-generated.c in src-generated, since we've explicitly told vpath to
include src-generated in its search path for .c files.  However, the vpath
command in GNU make checks to make sure that the directory that is being
added to the vpath exists before it actually adds it, and it silently
ignores the directory if it does not.  Since src-generated does not exist
when the Makefile is read in (it would only exist after the
main-generated.c file was generated), it doesn't end up in the vpath.

Thus we end up with a chicken-and-egg situation: the commands for building
src-generated/main-generated.c need to be invoked in order to create the
src-generated directory, but the src-generated directory must exist
already before the commands for building src-generated/main-generated.c
would be invoked (because vpath will not locate
src-generated/main-generated.c unless the directory exists).

One way to solve this problem is to change the vpath command to the
following:

vpath %.c src-generated $(shell mkdir -p src-generated)

This forces GNU make to create the src-generated directory before
evaluating the vpath command itself and since the src-generated directory
then exists when the vpath command is evaluated, it gets added to vpath
and everything works fine.  Observe the results of running GNU make after
making this change to the Makefile:

$ make test
mkdir -p src-generated
cp src/main.c src-generated/main-generated.c
gcc -o test src-generated/main-generated.c

So this provides a reasonable workaround for the problem.  However, it's
not ideal:

* It creates the src-generated directory every time the Makefile is run,
regardless of whether it was actually needed by any of the targets
* It is non-intuitive; anyone who hasn't searched for the solution to this
problem from the archives, or spent some time experimenting on their own
and discovering that vpath silently ignores nonexistent directories, will
not know how to solve this problem

My question is then, why not fix this in GNU make?  Why not remove the
prerequisite for the existence of the directory at the time that the vpath
command is evaluated?

If the check were taken out, then vpath would be exactly the set of paths
that the Makefile had specified, regardless of whether they existed or
not.  This would make my example work correctly but would mean that anyone
who depended upon vpath not including paths that don't exist would have a
problem.  I've thought about it and I haven't thought of any way that this
behavior (of vpath not including directories which don't exists) is
useful.  Can anyone think of any reason that this behavior needs to be
retained?

If not, I'd like to submit that we should "fix" GNU make with respect to
this issue.  I'd be happy to dig into the GNU make code and make the
change and submit the patch.  But before that, I'd like to hear others'
thoughts on this issue.

Thank you, and best wishes,
Bryan

------------------------------------------------------------------------
Bryan Ischo               address@hidden          2002 Honda CBR 600F4i
Mountain View, CA, USA    http://www.ischo.com     RedHat Fedora Core 3
------------------------------------------------------------------------