new feature idea: ingesting processed rulesets

[Zoltán Turányi]

Hello,

 

I have looked through the mail archives and the manual (to some degree of thoroughness) and have not found this idea yet. If, it exists, I apologize for the spam in advance.

 

I use make with autotools in multiple directories and have observed that parallel builds are limited to each directory, as autotools invoked make separately for each directory. This means that if one of the targets in a directory takes a long time to produce, many of the cores sit idle and the next directory does not get processed. Of course, this is needed sometimes, since the next directory may depend on the one being make’d. However, 1) most of the time this is not the case and 2) even if it were, these dependencies may actually be precisely known (autotools tracks these dependencies well). The problem, of course can be solved, by having a single Makefile for the many directories with a single invocation of make, but that is not always practical or easy.

 

So here is the idea. What if a subsequent invocation of make (in a subdir)– instead of building the target it is given – would just parse the makefile, create a full ruleset internally and inject this ruleset into the parent make’s ruleset. Then the subsequent invocation can simply return and let the parent instance of make actually complete the build. This would enable correct parallelization with full respect of the dependencies. This can, of course only be done if the subsequent invocation of make is the last thing its action is doing. (Or one can somehow split the action to run the remaining part after the subsequent make’s target is complete.)

 

Of course, this is not possible if the subdirectory creates artifacts on which subsequent directories depend, but without expressing that dependency. As far I am concerned, this is bad practice, but in this case we should simply disable this behaviour.

 

One way to implement this idea would be the parent make to open a pipe and pass its name to subsequent invocations of make via a command-line switch. Subsequent invocation would then dump the parsed rulesets into that pipe and exit. The ruleset, of course can contain environment variables and other context that may be different from that of the parent – this has to be taken care of. I do not know anything about how make represents the complete ruleset internally, so I have no idea, how complex this feature would be.

 

For example, let us assume we have 3 Makefiles, one main and one in a subdirectory and Makefile and sub1/Makefile and sub2/Makefile

 

Makefile contains:

 

all: subs main.o

                …link with sub1/lib1.o sub2/lib2.o

 

main.o: main.c sub1/lib1.h sub2/lib2.h

                …compile

 

subs:

                cd sub1

                make

              cd ../sub2

              make

 

 

While sub{1,2}/Makefile would contain

 

all: lib{1,2}.o

 

sub{1,2}.o: sub{1,2}.c sub{1,2}.h

                …compile

 

 

 

If we invoke make on the top makefile, it will compile the two lib{1,2}.o object files serially. With the proposed feature, the two invocations of make in the “subs” target would return immediately, by simply injecting their rules into the top make’s ruleset. The top make would correctly figure out that in order to do “subs” it has to build the “all” of both subdirs and would begin linking only after that.

 

Any comments? Is this a good idea? Can it be done?

 

Thanks,

Zoltan