Brittle -includes

[David Deutsch]

Hi everybody,

I've put together a somewhat extensive build system on top of make which
relies heavily on generating and including Makefiles (partly to avoid
The Dreaded Recursive Make). The centerpiece is a complement to %-rules
where I have blueprints for certain filetypes that I match against a
path via regex in a custom script that writes an actual makefile. I
guess it's my way of having %-rules with multiple, named % variables
across a path, allowing you do do things like:

---- ---- ---- ----
target/{{ path }}-w{{ width }}-h{{ height }}.jpg: source/{{ path }}.jpg
    cat $< | convert -resize {{ width }}x{{ height }} - jpg:- > $@
---- ---- ---- ----

I use it to build trees of linked documents and assets, so there is a
lot of overlap in the assets across documents. My approach is loosely
based on the patterns laid out in "Auto-Dependency Generation" [0].

A simplified version of the .mk file printer recipe is:

---- ---- ---- ----
$(makedir)/%.mk:
    $(mybin)/mkautoprint $* > $@
---- ---- ---- ----

When such an .mk file is created, it also includes dependencies of
further to-be-created .mk files like so:

---- ---- ---- ----
THEFILE: PATHOFMYDEPENDENCY
    $(mybin)/buildfile $*

# This .mk includes no recipe for PATHOFMYDEPENDENCY, so the script
attaches:

ifeq (,$(findstring $(makedir)/PATHOFMYDEPENDENCY,$(MAKEFILE_LIST)))
-include $(makedir)/PATHOFMYDEPENDENCY.mk
endif
---- ---- ---- ----

(The MAKEFILE_LIST ifeq is used to suppress warnings because multiple
documents could be trying to build the same asset)


My problem is that I keep running into situations where an .mk is
-include'd as seen above, but make never attempts to... make it,
resulting in a 'No rule to make target' error. I suspect there is some
kind of race condition going on where it does "see" the include, but
missing dependency error is triggered before that can occur. This
happens consistently, no matter whether I use -j or not.

The odd thing is that it will happily build hundreds of those .mk files
and then fail at one of them. You'd think that it's due to dependencies,
but generating the .mk itself has none. Similarly, when I remove one of
those .mk files, make often does not remake them on the next turn.

Does anybody have a tip on how I could debug this? Is there a better
approach for this strategy that I could look into? It's quite likely
that I missed something when reading "Auto-Dependency Generation". Even
after years of using make heavily, now, I still can't answer questions
like "do i need to use include or -include?" with absolute certainty. (I
think I got that bit from "The GNU Make Book" by John Graham-Cumming and
just cargo-cult used it for months now.)

I hope I explained the problem well enough - do let me know if there is
anything I need to further clarify!

regards,
David


[0]
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/