Re: [RFC]serialize the output of parallel make?

[Howard Chu]

Paul Smith wrote:
> On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote:
> > As parallel make are becoming more and more popular,  can make
> > serialize the output of parallel make?
> >
> > Make can redirect every parallelly issued shell's output to an
> > temporary file,  and output the stored output serially, as if in a
> > serial make.
>
> This would be a good thing, but as always the details are not quite so
> trivial.

Aside from the difficulties outlined below, I just am not fond of having 
output batched up instead of appearing in realtime. It tends to complicate the 
polling logic too (though I guess in this case, you just have to cat the 
appropriate file[s] whenever a child process ends.)

The scheme that I grew up with on Alliant Concentrix was just to prefix each 
output line with its job number "|xx|blah blah blah". It obviously requires a 
pipe for each child process' output, so that lines can be read by the parent 
make and then the prefix attached. In the original jobserver prototype I used 
unique bytes for each job token so that the token == the job ID, with an eye 
toward adding this support later. But that obviously limited it to supporting 
only 256 concurrent jobs, and these days we already get complaints that it's 
limited to only 4096. Using a pipe per job would likewise cut make's maximum 
job count in half (or worse, if using a separate stderr pipe).

I still favor this latter approach because it keeps the output flowing in 
realtime, and its easy enough to use grep if you want to zero in on a single 
output stream. But the cost in resources will add up...
> We have to ensure that these temporary files are cleaned up properly,
> even in the face of users ^C'ing their make invocations.  We also need
> to verify that whatever methods we use will work properly on Windows and
> VMS and other operating systems make supports (where are their "/tmp"
> equivalents?)
>
> And, what about stdout vs. stderr?  Should we write both to the same
> file?  Then we lose the ability to do things like "make -j4 2>/dev/null"
> since all output will be written to stdout (presumably).  Or should we
> keep two files per command, one for stdout and one for stderr?  But
> that's even worse since then when we printed it we'd have to print all
> the stdout first then all the stderr, which could lose important
> context.
>
> Then there's the possibility that some commands will behave differently
> if they are writing to a TTY, then they will if they're writing to a
> file.  Do we not care about that, or do we try to do something crazy
> with PTYs or similar (ugh!)
>
> And of course we have to have a guaranteed unique naming strategy in
> case multiple instances of make are running on the same system at the
> same time, maybe running the same makefiles and even building the same
> targets.  On POSIX systems we can use tmpfile() or mkstemp() or
> something but other systems might need other measures.
>
> These are just some things I thought of off the top of my head.
>
> It certainly does not mean that it would not be a good thing to have
> this ability though.


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/