[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: feature request: parallel builds feature

From: Howard Chu
Subject: Re: feature request: parallel builds feature
Date: Tue, 30 Apr 2013 18:55:30 -0700
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0 SeaMonkey/2.20a1

Jim Michaels wrote:

I wasn't digressing.  I was explaining the point.  the concept I am trying to
present as a solution to the problem of making parallel stdin for --jobs in
gnu make (which currenty doesn't work and is I guess single-threaded) is to
make a separate terminal or command shell for each job, such as via a
generated batch file or shell script.

this is as simple as I can make it.

Who said stdin was a problem? Fundamentally the jobs spawned by make are batch jobs - they should not be requesting input from stdin in the first place.

Semantically the effect of running parallel make must be identical to running serial make. You cannot guarantee this to be true if jobs are reading from stdin because stdin's supply of data is inherently serial but the order it gets read is non-deterministic in a parallel build.

If the jobs you're spawning from make require input from stdin you need to rewrite those jobs.

at the end of the shell script, you can put in whatever you like, such as
synchronization stuff saying "I am done" by creating  a semaphore file, a flag

the problem would then be porting BASH to windows and other platforms in order
to handle --jobs.

bash has already been ported to Windows.

I have learned that on a machine with 12 threads and 64GB of memory, you can
have 50+ jobs running.

    *From:* Paul Smith <address@hidden>
    *To:* Jim Michaels <address@hidden>
    *Cc:* address@hidden
    *Sent:* Monday, April 22, 2013 10:56 AM
    *Subject:* Re: feature request: parallel builds feature

    On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote:
     > it currently has a problem with stdin, because at this point there is
     > only one of those, only 1 of them gets it, and the others starve. so
     > if your build needs stdin or creates files from the commandline using
     > heredocs, you can't use it (check first!). you will get an error. gnu
     > has not yet figured out a solution yet (I have, multiple shells IF you
     > can control them... probably can't without some work creating batch
     > files for the jobs). so there is a solution. even with a batch file,
     > make would need some sort of way of reporting back error conditions. I
     > think there are ways of managing that with files via presence-detect,
     > sort of like semaphores. they should get cleared when the job ends, or
     > when a switch is given to clear the state for that session if the
     > session was broken with ctrl-c. well, I suppose a ctrl-c handler
     > should still kill those terminals or cmd shells and clear up those
     > files.
     > what do you think?
     > if a terminal is opened, it should be created without a window. some
     > OS's have that option. some don't, like freeDOS, which would not have
     > the ability to do terminals, parallel shell windows, or even the
     > --jobs feature (but that's implementation-dependent).

    Please keep the mailing list CC'd.  Thanks.

    I'm afraid I still don't understand what you're asking for here.  You'll
    need to back up and provide a description of your needs in a clear,
    orderly way without digressions.

    Yes, it's true that GNU make only provides its stdin to one job at a
    time and which job gets it is essentially random.  In order to address
    this we'd need to see a specific use-case or requirement, but my
    suspicion is that all such possible use-cases are better solved by a
    change of process at a level above what make can provide.

Bug-make mailing list

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

reply via email to

[Prev in Thread] Current Thread [Next in Thread]