Re: [RFC]serialize the output of parallel make?

[Edward Welbourne]

>>> Make can redirect every parallelly issued shell's output to an
>>> temporary file,  and output the stored output serially, as if in a
>>> serial make.

+1 for wanting this as a make feature.
We a hack in some of our makefiles to implement essentially exactly
the above.  For reference, here's the essence of a kludge known to
work:

EACHOUT = address@hidden
ALLOUT  = $(GENROOT)/build_log.out
LOGEACH = ; flock $(ALLOUT) \
        sh -c 'echo "$1 $<)" | cat - $(EACHOUT) >> $(ALLOUT)'; \
        $(RM) $(EACHOUT)

Get every command whose output you care about to redirect to
$(EACHOUT); at the end of each recipe that's doing any of that (in
many cases this'll be right after the redirect, but some command-lines
may be fiddlier) append $(LOGEACH).  That's assuming you combine
stdout and stderr; if not, you need to do similar for EACHERR and
ALLERR.  For bonus points, add an || failed=yes to each command in
each affected recipe and a ; [ -z "$$failed" ] to the end of LOGEACH,
so that make knows which recipes fail.

>> And, what about stdout vs. stderr?
Yup, there are endless problems with this.
On the other hand, there are situations that really do beg for it.

In particular, automated build systems are an important part of many
development methodologies; obviously, using -j in them is desirable;
and getting coherent logs out of them, in which each compilation
unit's errors and warnings appear as a single block, is crucial.
These typically don't even bother with tee, they just >output.txt 2>&1
directly.

The results need to be tidy, because they are typically parsed by
something that converts them to HTML, needs to make the errors and
warnings stand out and may also wants to compare warnings against
those from earlier builds in order to flag them as regressions.  Such
parsing gets severely messed up when output from several commands gets
interleaved.

> The scenario like "make -j4 2>/dev/null" may be very rare, but
> scenario like "make -j4  2>&1 |  tee output.txt" may be common.
I concur.
Those discarding make's output probably discard all of it, not
selectively stderr.

> When user of make provide the optional output-serializing command line
> option, he known what he need, he is responsible for the mess
> situation.

Making this an optional feature makes sense, particularly given the
myriad complications Paul points out.  I suggest options along the
following lines: <sketch>

 --demux-stdout[=filename]
 --demux-stderr[=filename]
 --demux-outerr[=filename]

        When using -j, capture each recipe's standard output, standard
        error or both (interleaved as by 2>&1) and emit it atomically
        (i.e. not interleaved with the corresponding output of any
        other recipe), optionally to filename.

        (Even when -j is not used, --demux-stdout and --demux-stderr
        can be used to separate the standard output and error streams
        of commands.)

        Note that some commands may behave differently if output is
        not to a terminal (as will be the case if these flags are
        used).  Some commands may emit output that only makes sense
        when stdout and stderr are interleaved in the order in which
        they were output.  Use at your own peril.

</sketch> Quite how implementable that is, I leave Paul to decree,

        Eddy.