Re: feature request: parallel builds feature

[Jim Michaels]

what if you in your makefile are creating files from scratch using echo, based on system configuration information?
I know I have to do that in order to create XML manifest files for resources to compile and link in via resource compiler for windows builds.

I can give you a sample of some of the lines I have to convert from .cmd batch files as they are now, to makefiles as the system will be eventually when I get the migration completed.


    echo -----creating 32-bit 9x+ manifest...  don't change this particular block or nothing will work.
    echo ^<?xml version="1.0" encoding="UTF-8" standalone="yes"?^>>32\%outfile%.%extension%.manifest
    echo ^<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"^>>>32\%outfile%.%extension%.manifest

rem it works with description on XP, but I don't know about other platforms, so I am leaving it out.
rem     echo     ^<description^>%manifest_apptitle%^</description^>>>32\%outfile%.%extension%.manifest

    echo     ^<ms_asmv2:trustInfo xmlns:ms_asmv2="urn:schemas-microsoft-com:asm.v2"^>>>32\%outfile%.%extension%.manifest
    echo         ^<ms_asmv2:security^>>>32\%outfile%.%extension%.manifest
    echo             ^<ms_asmv2:requestedPrivileges^>>>32\%outfile%.%extension%.manifest
    echo                 ^<ms_asmv2:requestedExecutionLevel level="%manifest_executionlevel%"^>>>32\%outfile%.%extension%.manifest
rem it works on XP, but I don't know about other platforms, so I am leaving it out.
rem     echo                 ^<ms_asmv2:requestedExecutionLevel level="%manifest_executionlevel%" uiAccess="%manifest_uiaccess%"^>>>32\%outfile%.%extension%.manifest
    echo                 ^</ms_asmv2:requestedExecutionLevel^>>>32\%outfile%.%extension%.manifest
    echo             ^</ms_asmv2:requestedPrivileges^>>>32\%outfile%.%extension%.manifest
    echo         ^</ms_asmv2:security^>>>32\%outfile%.%extension%.manifest
    echo     ^</ms_asmv2:trustInfo^>>>32\%outfile%.%extension%.manifest
    echo ^</assembly^>>>32\%outfile%.%extension%.manifest

    echo -----creating 64-bit 9x+ manifest...  don't change this particular block or nothing will work.
    echo ^<?xml version="1.0" encoding="UTF-8" standalone="yes"?^>>64\%outfile%.%extension%.manifest
    echo ^<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"^>>>64\%outfile%.%extension%.manifest

rem it works with description on XP, but I don't know about other platforms, so I am leaving it out.
rem     echo     ^<description^>%manifest_apptitle%^</description^>>>64\%outfile%.%extension%.manifest

    echo     ^<ms_asmv2:trustInfo xmlns:ms_asmv2="urn:schemas-microsoft-com:asm.v2"^>>>64\%outfile%.%extension%.manifest
    echo         ^<ms_asmv2:security^>>>64\%outfile%.%extension%.manifest
    echo             ^<ms_asmv2:requestedPrivileges^>>>64\%outfile%.%extension%.manifest
    echo                 ^<ms_asmv2:requestedExecutionLevel level="%manifest_executionlevel%"^>>>64\%outfile%.%extension%.manifest
rem it works on XP, but I don't know about other platforms, so I am leaving it out.
rem     echo                 ^<ms_asmv2:requestedExecutionLevel level="%manifest_executionlevel%" uiAccess="%manifest_uiaccess%"^>>>64\%outfile%.%extension%.manifest
    echo                 ^</ms_asmv2:requestedExecutionLevel^>>>64\%outfile%.%extension%.manifest
    echo             ^</ms_asmv2:requestedPrivileges^>>>64\%outfile%.%extension%.manifest
    echo         ^</ms_asmv2:security^>>>64\%outfile%.%extension%.manifest
    echo    ^</ms_asmv2:trustInfo^>>>64\%outfile%.%extension%.manifest
    echo ^</assembly^>>>64\%outfile%.%extension%.manifest



there is more. there are if statements involved, etc.
currently, there is no manifest tool in the compiler set for mingw-w64 or mingw. nothing is planned.

---

From: Howard Chu <address@hidden>
To: Jim Michaels <address@hidden>; "address@hidden" <address@hidden>
Cc: "address@hidden" <address@hidden>
Sent: Tuesday, April 30, 2013 6:55 PM
Subject: Re: feature request: parallel builds feature

Jim Michaels wrote:
>
> I wasn't digressing.  I was explaining the point.  the concept I am trying to
> present as a solution to the problem of making parallel stdin for --jobs in
> gnu make (which currenty doesn't work and is I guess single-threaded) is to
> make a separate terminal or command shell for each job, such as via a
> generated batch file or shell script.
>
> this is as simple as I can make it.

Who said stdin was a problem? Fundamentally the jobs spawned by make are batch
jobs - they should not be requesting input from stdin in the first place.

Semantically the effect of running parallel make must be identical to running
serial make. You cannot guarantee this to be true if jobs are reading from
stdin because stdin's supply of data is inherently serial but the order it
gets read is non-deterministic in a parallel build.

If the jobs you're spawning from make require input from stdin you need to
rewrite those jobs.

> at the end of the shell script, you can put in whatever you like, such as
> synchronization stuff saying "I am done" by creating  a semaphore file, a flag
> file.
>
> the problem would then be porting BASH to windows and other platforms in order
> to handle --jobs.

bash has already been ported to Windows.
>
> I have learned that on a machine with 12 threads and 64GB of memory, you can
> have 50+ jobs running.
>
>
>    ------------------------------------------------------------------------------
>    *From:* Paul Smith <address@hidden>
>    *To:* Jim Michaels <address@hidden>
>    *Cc:* address@hidden
>    *Sent:* Monday, April 22, 2013 10:56 AM
>    *Subject:* Re: feature request: parallel builds feature
>
>    On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote:
>      > it currently has a problem with stdin, because at this point there is
>      > only one of those, only 1 of them gets it, and the others starve. so
>      > if your build needs stdin or creates files from the commandline using
>      > heredocs, you can't use it (check first!). you will get an error. gnu
>      > has not yet figured out a solution yet (I have, multiple shells IF you
>      > can control them... probably can't without some work creating batch
>      > files for the jobs). so there is a solution. even with a batch file,
>      > make would need some sort of way of reporting back error conditions. I
>      > think there are ways of managing that with files via presence-detect,
>      > sort of like semaphores. they should get cleared when the job ends, or
>      > when a switch is given to clear the state for that session if the
>      > session was broken with ctrl-c. well, I suppose a ctrl-c handler
>      > should still kill those terminals or cmd shells and clear up those
>      > files.
>      > what do you think?
>      > if a terminal is opened, it should be created without a window. some
>      > OS's have that option. some don't, like freeDOS, which would not have
>      > the ability to do terminals, parallel shell windows, or even the
>      > --jobs feature (but that's implementation-dependent).
>
>    Please keep the mailing list CC'd.  Thanks.
>
>    I'm afraid I still don't understand what you're asking for here.  You'll
>    need to back up and provide a description of your needs in a clear,
>    orderly way without digressions.
>
>    Yes, it's true that GNU make only provides its stdin to one job at a
>    time and which job gets it is essentially random.  In order to address
>    this we'd need to see a specific use-case or requirement, but my
>    suspicion is that all such possible use-cases are better solved by a
>    change of process at a level above what make can provide.
>
>
>
>
>
>
> _______________________________________________
> Bug-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/bug-make
>


--
  -- Howard Chu
  CTO, Symas Corp.          http://www.symas.com
  Director, Highland Sun    http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/