RE: Parallel Build with GNU make

[Martin Dorey]

I'm not sure that I'll be adding anything that isn't a statement of the obvious 
but I can explain how we solve this in our environment.  We have the subfolders 
specified in a variable, SUBDIRS.  Another variable, 
BUILD_RECURSION_PREREQUISITES is synthesized from SUBDIRS to contain the like 
of recurse.%.subfolder1 recurse.%.subfolder2 etc.  BUILD_TARGET_STEMS contains 
phony target names that a user might pass to make, like "release" and "debug".  
These stems eventually form the % in those prerequisites, with something like:

$(BUILD_TARGET_STEMS): %: $(BUILD_RECURSION_PREREQUISITES)

Then I serialize the recursion, where appropriate, with this gruesome fragment:

JOIN_ADJACENT_PAIRS = $(filter-out @@:% %:@@,$(join $(addsuffix :,$(1) @@),@@ 
$(1)))
CONSTRUCT_DEPENDENCY_SERIALIZATION_RULES = $(foreach STEM,$(1),$(subst 
%,$(STEM),$(call JOIN_ADJACENT_PAIRS,$(2))))
SERIALIZE_DEPENDENCIES = $(foreach RULE,$(call 
CONSTRUCT_DEPENDENCY_SERIALIZATION_RULES,$(1),$(2)),$(eval $(RULE)))
$(call 
SERIALIZE_DEPENDENCIES,$(BUILD_TARGET_STEMS),$(BUILD_RECURSION_PREREQUISITES))

-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Roland Schwingel
Sent: Monday, February 01, 2016 23:40
To: Eli Zaretskii; address@hidden
Subject: Re: Parallel Build with GNU make

Sorry for following up on my own post,but there has been a "bug" in the 
last sentence... See for correction below

Am 02.02.2016 um 08:35 schrieb Roland Schwingel:
> Hi...
>
> Eli Zaretskii wrote on 01.02.2016 20:14:17:
>
>  > Add "--debug=j" to the make command-line switches and see if it
>  > launches more than one command at a time.
>  >
>  > Anyway, your makefile.mak file can be written in such a way as to
>  > effectively prohibit parallelism, due to how it lays out dependencies.
> This is something I am struggeling on. I tried this a couple of times
> but could not make it working in a way that is satisfying for me, as I
> could not find the correct way to control parallelism. I am aware of
> .NOTPARALLEL but this might not be enough for me.
>
> I have this examplarily layout:
>
> subfolder1 - subfolder4 - sub4_1.c
>               sub1_1.c     sub4_2.c
>               sub1_2.c
>               ...
> subfolder2 - sub2_1.c
>               sub2_2.c
> subfolder3 - sub3_1.c
>               sub3_2.c
> main1.c
> main2.c
> main3.c
>
> All subfolders shall be visited recursively first. No parallelism is
> allowed in visiting folders. But when in a folder and it comes to
> concrete compiling parallelism should be allowed.
>
> This should result in this compileorder for the above example
> Visit subfolder1 (not in parallel)
> Visit subfolder4 (not in parallel)
> Compile sub4_1.c and sub4_2.c in parallel
> Return to subfolder1
> Compile sub1_2.c and sub1_2.c in parallel
> Return to main
> Visit subfolder2 (not in parallel)
> Compile sub2_1.c and sub2_2.c in parallel
> Return to main
> Vist subfolder3 (not in parallel)
> Compile sub3_1.c and sub3_2.c in parallel
> Return to main
> Compile main1.c,main2.c and main3.c in parallel
>
> When doing a make -j 4 for instance make is visiting all subfolders in
> parallel which is not desired. When I am using .NOTPARALLEL the whole
> subfolder is not compiled in parallel which is also not desired. Each
> folder visit is a concrete submake in my case. Each folder has its own
> makefile
>
> Any clue on how to achieve the flow I want to have? Visiting folders non
> recursive while compiling itself inside of the folders is recursive?
Should read:
Visiting folders non PARALLEL while compiling itself inside of the 
folders is IN PARALLEL.

Sorry and thanks again for your help,

Roland



_______________________________________________
Bug-make mailing list
address@hidden
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_mailman_listinfo_bug-2Dmake&d=CwICAg&c=DZ-EF4pZfxGSU6MfABwx0g&r=oBMzc8Omr1YTgjig4n4076T3IKL7TuNH9HpVbojD-ms&m=MfFKkv-mCbvJA9yNJWbqRoqitBf5hjsRRy1xBUq8peA&s=Rf15u8uDSCGNGENESJhWzqBOa086GoGTrfTFimBzepg&e=