Re: parallel make help

[Bob Friesenhahn]

On Thu, 10 Apr 2008, Bob Rossi wrote:
> If I moved everything to a single Makefile.am at the top level, would
> that improve the speed of things so that it could continue to build
> in parallel?

Yes.  If you convert your project to a non-recursive build, then it is 
very likely that considerably more parallelism will be exposed.

In any large build, there are certain points where all parallism must 
cease and it is necessary to wait for one task to complete. 
Typically this occurs whenever there is a link of a library that other 
targets need. Libtool convenience libraries are one of these points. 
It also occurs whenever a directory is entered via the simplistic 
SUBDIR mechanism.  With the simplistic SUBDIR mechanism, only targets 
built in the subdirectory being built are candidates for parallel 
compile. With a non-recursive build you can eliminate use of the 
SUBDIR mechanism and convenience libraries.

Depending on how you structure your libraries and apportion source 
modules between common libraries and final executable code, you can 
achieve massive parallism with a non-recursive build.

Users with multiple cores and lots of memory will be happy since there 
is almost linear speedup.  It remains to be seen if distributing 
compile jobs to other machines speeds things up since there is a cost 
associated with distributing compile jobs.  If your source modules are 
quick to compile, there may not be any win.

> If it is in fact faster, like I'm imagining it might be, is there any
> way to make the "Makefile.am in each subdir" approach be just as fast?

No.  However, you can still gain the maintenance benefits of 
"Makefile.am in each subdir" by using the Automake include mechanism 
with non-recursive build.

Bob
======================================
Bob Friesenhahn
address@hidden, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/