Re: How do parallel builds scale?

[Howard Chu]

Ludovic Courtès wrote:
> Hello,
>
> There was recently discussion as to how configure could be parallelized.
>
> I ran a series of build time measurements on a 32-core machine, with
> make -jX, with X in [1..32], and the results are available at:

It's been my experience that compile-oriented builds tend to be I/O bound and 
generally have excess CPU capacity in the range of 30-40%. As such I always 
use make -jX where X is number of cores * 1.5, which tends to keep the idle 
CPU percentage below 5%.

>    http://hubble.gforge.inria.fr/parallel-builds.html
>
> There are also insightful per-package details:
>
>    http://hubble.gforge.inria.fr/parallel-build-details.html
>
> There are packages whose configuration phase is noticeably longer than
> the build time.

Yes. I've seen several projects where the autotool files are larger than the 
actual program source code. IMO, any time your build overhead takes up more 
than 1% of your total project size, you've done something wrong.

> OTOH, packages with a long build time (with a sequential time in
> boldface) don’t scale well either, but that is not configure’s fault.

If only more people would learn to write proper Makefile dependencies...

http://highlandsun.com/hyc/#Make

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/