Re: Thoughts on limiting the parallel build load, suggestion for a new "-j auto" option

[Edward Welbourne]

> Go and shoot the moron who wrote that script.

Easy to say, but no use to someone who needs to deploy a package whose
build infrastructure includes a badly-written script.  If the author of
that script has been shot, there's probably no-one alive who understands
how to build the package.  Quite possibly, the author only knocked the
script together for personal use, but made it available to the project
as a start-point for others to set up their own builds, only to see it
chucked into the distribution because it was more use than nothing.

> -l is utterly useless.

Hyperbole.

> Load average is computed too slowly; by the time it passes any useful  
> threshold the actual make load will have spiralled out of control.

The -l value is not as useful as we might like, for this and other
reasons, but - in practice - in conjunction with -j, it makes it
possible to tune make's behaviour so that I can make good use of my CPU
without rendering the machine unusable while waiting for a build.

> The make -j value should never be encoded in any file; it should only  
> ever be set by the user invoking the actual top level make command.

and therein lies the problem with -j and -l; they require the user to
have a sophisticated understanding of the system parameters and build
load profile in order to guess a half-way decent value for them.  Such
options are much less useful than ones that have sensible defaults.

For developers of the package being built, it's possible to learn - by
trial and error - what settings tend to work out reasonably sensibly on
specific machines.  I have heard various ad hoc rules of thumb in use,
typically number of CPUs plus either a fixed offset or some fraction of
the total number of CPUs, as either -j or -l value.  In the end, I know
that my old machine grinds to a halt if I let make take load > about 4
and my shiny new machine accepts -j 8 -l 12 without breaking a sweat -
and gets the build done in a few minutes, instead of half an hour.

For anyone who has to build a package that isn't the primary business of
their work - for example, a distribution maintainer, for whom each
package is just one in a horde of many - configuring the right -j and -l
flags for each run of make is not practical.  It would make sense to
provide them with a sensible way to tell make to make good use it can of
the CPUs available, without bringing the system to its knees.

The present unconstrained -j behaviour is, in any case, self-defeating.
It starts so many processes that the time the machine spends swapping
them all out and back in again swamps the time they spend actually doing
anu useful work.  The build would complete sooner if fewer processes
were started.

I think R. Diez's suggestions are a constructive step towards designing
some half-way sensible heuristics - those with better understanding of
what make's internal decision-process looks like can doubtless improve
on them, which I'm sure is R. Diez's hope and intent in starting this
discussion.  We don't need -j auto to achieve perfectly optimal tuning
on every machine; it'll be good enough if it can do builds faster than
make's default implicit -j 1, without (too often) loading the machine so
heavily as to interfere with browsing, reading mail, playing nethack and
other things developers like to still be able to do while waiting for a
build.

        Eddy.