Re: Some serious issues with the new -O option

[Paul Smith]

Eli Zaretskii <address@hidden> writes:
> > If you want different behavior you can change your rule to use "+" on
> > the two echo lines, so that they're also considered recursive and not
> > saved up.
> 
> If I do that, the echo from rec1 and rec2 mix up:
> 
>   D:\gnu\make-3.82.90_GIT_2013-05-01>gnumake -f mkfsync3 -j -O
>   start rec2start <<<<<<<<<<<<<<<<<<<<
>    rec1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>   gnumake -f mkfsync simple
>   gnumake -f mkfsync simple
> 
> Is this also expected?

You're right, I'm wrong.  Using "+" is not really like -Ojob, because
with -Ojob we still sync the output of each line.  Using "+" the job
just sends the output directly to the terminal with no sync.  Thus this
mixing up is what I'd expect (same as not using -O at all).

I begin to wonder if this really requires a new per-line special prefix
character like "@", "+", "-", that controls the syncing of that
particular line.  I'm very reluctant to go there as it is a BIG change
and a backward-compat issue.  Also it seems that unlike the existing
prefix characters, for this one we'd have as much need for a way to turn
OFF sync as to turn it ON... bleah.


On Thu, 2013-05-02 at 05:53 +0300, Eli Zaretskii wrote:
> > Now, if you do nothing special for recursive make, you'll get no output
> > from the entire build until it is completely done, because all the
> > output from the recursive make command is going to the temporary file
> > for that target, then it all gets dumped at the same time.
> 
> Not every Makefile looks like that on its top level.  I agree that we
> should cater to the above, but perhaps we could do that without
> "punishing" the other use cases.  For example, perhaps we should have
> a -Osub-make option that will _not_ activate sync-output on the top
> level, only in the sub-make's.  This should produce the desired
> results, I think.

Can you clarify what the "desired results" are?  I seem to have lost the
thread.  What is the behavior you see now that you are trying to avoid
(or you don't see that you're trying to achieve)?

Capture of the sub-make will mean that the entire output of that
sub-make, and all of its recipes including ITS sub-sub-makes, will be
sent to a temporary file and not displayed on the screen until the
entire sub-make is completed.  In what situation would we want to choose
this, regardless of level of sub-make?

In general I see no benefit in trying to special-case any particular
level of make.  For some builds the top level does all the work.  For
some the second level.  For some the third.  For many, different levels
for different parts of the same build.

> > I don't understand the change that you're suggesting.  That's exactly
> > what -Omake does today: whenever any recipe finishes it is flushed to
> > the screen as a single unit, and no special handling is given to
> > recursive makes.
> 
> In my case, I see all the output at once.  Maybe I misunderstand what
> -Omake is supposed to do, too.

I think you and I said the same thing: the output from recursive makes
is saved up and flushed all at once...?

Tim Murphy <address@hidden> writes:
> One optimisation I have thought of in the past for this situation
> would be to allow a single "job"  to hold onto the lock when it
> obtained it.
> 
> This way it could output directly to the console while all other jobs
> would have to buffer. When it released, the next job lucky enough to
> grab the lock might have a full buffer already.
> 
> It might appear to be less choppy.  Not sure how it would perform.

It might be less choppy (or it might not: it depends on your targets:
are they all more-or-less equally chatty?) but we discussed this
possibility and decided that it would be too costly in terms of
performance.

All you need is for one long-running job to get the terminal and pretty
soon it's the only job running as all others have finished but can't
continue with the next one until they can grab the terminal and dump
their output.  Personally I think this is a serious enough problem that
it's probably not even worth doing the work in GNU make to provide this
as an option.  I suspect people would probably much rather just not use
-O and live with interleaved output, or use -O and live with some choppy
output, than suffer essentially random increases in build times.  I
could be wrong though.