Re: Using shell wrapper for descrambling parallel make output

[Atte Peltomäki]

On Fri, Nov 11, 2011 at 10:24:25AM -0500, David Boyce wrote:
> On Fri, Nov 11, 2011 at 9:26 AM, Atte Peltomäki <address@hidden> wrote:
> > Hi,
> >
> > As you know, running parallel builds with '-jX' makes the shell output
> > difficult to read, since output from parallel jobs are mixed. To remedy
> > this, the use of a buffering shell wrapper has been suggested:
> >
> > http://cmcrossroads.com/cm-basics/12838-descrambling-parallel-build-logs
> >
> > I liked the idea, but implementation was lacking so I rewrote the
> > wrapper.
> 
> That's a good thing to do but the work may be duplicative. See
> https://github.com/boyski/SYNCSH. Take a look and tell me if you think
> it's worth comparing notes in order to get the best of both.
> 
> > If I have time and interest, I will look into implementing the wrapper
> > functionality into make itself. Don't hold your breath, though.
> 
> Similarly, almost the same thing has already been done and a patch
> exists against 3.82 which I hope will make it into 3.83. See
> http://savannah.gnu.org/bugs/?33138. There's also a long thread in the
> bug-make archives from around the same time (4/2011) discussing it.

Nice work. Your implementation seems much more refined than mine. Only
one thing catches my attention; your version doesn't seem to properly
preserve the original line ordering between stdout and stderr. I suggest
solving this as I did:

- Don't write the output to files, just use a pipe, change vfork() to 
  fork() and read child output from the parent using the pipe.

- Use a simple struct as a buffer for the data: one member of the struct
  is an int that is 0 for stdout and 1 for stderr, other member is a one
  line of output (char *). 

- Use a temporary buffer to read() from pipe, then dynamically allocate
  only needed amount of memory per line to the the struct char* member.

See my code source for details.

-- 
Atte Peltomäki
     address@hidden <> http://kameli.org
"Your effort to remain what you are is what limits you"