I haven't seen similar issues but I have a hypothesis: make is single-threaded and therefore it consumes output from jobs in the same thread it uses to reap/spawn new jobs. If make is spending a large enough amount of time consuming output then this will impacts the rate at which it can spawn/reap new jobs.
I ran some tests to see when make attempts to consume output from child processes and here's what I learned:
(a) For jobs executing within a single instance of make (not recursive) it doesn't appear to begin consuming text until a recipe completes (I only tested this with --output-sync=target; I expect similar to be true of other sync modes excluding none).
(b) If you use --output-sync=target and a recipe contains the string ${MAKE} at all -- even if it doesn't actually start a recursive make instance -- the way it consumes output is different (more on that below)
(c) make will not stop reading job output until there's nothing left to consume. During that time it cannot reap/spawn new jobs. If your jobs produce a large enough amount of text this could create enough small delays that aggregate into the noticeable delays you're seeing.
As an example to demonstrate (b):
.RECIPEPREFIX := <
SHELL := bash
OUTER := $(shell echo -n {0..15})
INNER := $(shell echo -n {16..32})
all: outer
outer: ${OUTER}
inner: ${INNER}
${OUTER}:
<@${MAKE} -f /tmp/makefile inner
${INNER}:
<@perl -e 'my $$tmp = "${MAKE}"; map { printf "x\n"; sleep 1 } (0..100)'
I executed with (make 4.1):
make -j64 --output-sync=target
... and it begins printing output almost immediately in spite of the fact none of the recipes could have finished. If you take ${MAKE} out of the recipe then it behaves as expected.
Now consider what I said w/regard to (c): it will only move on to spawning/reaping other jobs (or consuming other inputs) if there's nothing to read for the job it's currently consuming text from.
When I use --output-sync=recurse I don't see the same behavior, but if your build recurses more than 1 level then perhaps each sub-make attempts to consume output from its recursive make calls, possibly eating away at time it could spend spawning/reaping jobs.
If I'm right this could be completely avoided for those of us who would rather have log files instead of having everything dumped to the terminal. If the parent make still needs to consume the output text, though, then the limiting factor is: make is single-threaded and consumes input from recursive jobs in the same thread it spawns/reaps new jobs.
-brian