Re: feature request: parallel builds feature

[Paul Smith]

On Tue, 2013-04-30 at 17:20 -0700, Jim Michaels wrote:
> I wasn't digressing.  I was explaining the point.  the concept I am
> trying to present as a solution to the problem of making parallel
> stdin for --jobs in gnu make (which currenty doesn't work and is I
> guess single-threaded) is to make a separate terminal or command shell
> for each job, such as via a generated batch file or shell script.
> 
> this is as simple as I can make it.

You need to give a concrete example of the problem you're trying to
solve.  When the manual discusses stdin it means that only one job at a
time can read from the make program's stdin.

Multithreading won't help because there is only one input to read from,
regardless of how many threads do the reading.

What this limitation is discussing is if there were a makefile like
this:

    read: read1 read2
    read1 read2: ; @echo $@: enter a word: ; read word ; echo $@: $$word

Both of these two targets read from stdin.  If you run them serially, it
works:

    $ make
   read1: enter a word:
   fooblatz
   read1: fooblatz
   read2: enter a word:
   barflatz
   read2: barflatz

If you run in parallel then both the read1 and read2 targets run at the
same time and both want input from stdin, at the same time.  There's no
way this can work: when you typed a word how could you know or specify
which read operation got it?  So make arbitrarily chooses one of the
jobs to get the input and the others have their stdin closed.

But if course, this doesn't impact in any way rules like this:

    read: read1 read2

    read1 read2: ; @word=`cat address@hidden; echo $@: $$word

Now if you have files like read1-input and read2-input, those will be
read inside these rules and behave properly.

> I have learned that on a machine with 12 threads and 64GB of memory,
> you can have 50+ jobs running.

This depends very much on what those jobs are doing.  Obviously you CAN
run as many jobs as you want.  However I've never heard of being able to
get more than #CPUs plus a few jobs running at the same time without
making the build _slower_.  At some point the kernel will simply thrash
trying to keep all those jobs running at the same time, if they
seriously outnumber the cores available to run them on.