Re: [bug #33138] .PARLLELSYNC enhancement with patch

[Frank Heckenbach]

Paul Smith wrote:

> On Thu, 2013-04-18 at 19:09 +0200, Frank Heckenbach wrote:
> > This mechanism was unaffected by my output-sync patch, and I
> > expected your change broke it.
> 
> I was reading your email with interest, waiting for the punch-line, but
> then after all that description you just said that the change broke it,
> with no explanation :-).  My question is, what about the new behavior
> does not work for you?

I said I *expected* your change broke it, but it didn't. :-)
Let me explain:

> The change I made was to have all the jobs in a target recipe write
> (appending of course) to the same temp file, then after the last job in
> the recipe is completed the output for the entire recipe is generated.
> So for example, if you have:
> 
>   foo bar:
>           : $@ one
>           : $@ two
>           : $@ three
> 
> and you run "make -j -O", the way you had it the output from each job
> would be mixed something like this (the actual result order will be
> slightly random):
> 
>   foo one
>   bar one
>   foo two
>   bar two
>   bar three
>   foo three

And with my progress mechanism, that's exactly what I want. In my
case it'd look like this:

[Start] Compiling foo.c
[Start] Compiling bar.c
# time passes
foo.c: some error
# time passes
bar.c: some error
# time passes
[End] Compiling bar.c
# time passes
[End] Compiling foo.c

This is useful (to me) because at any time, I know what's running.
("[Start]" messages minus "[End]" messages.)

> with my change you still get the same results but only after all jobs
> are complete, and they will be collected:
> 
>   foo one
>   foo two
>   foo three
>   bar one
>   bar two
>   bar three

In my case:

# time passes
[Start] Compiling foo.c
foo.c: some error
[End] Compiling foo.c
# time passes
[Start] Compiling bar.c
bar.c: some error
[End] Compiling bar.c

Here, I don't see what's running, since the start and end messages
always appear together, and I don't see a job started until it's
already finished.

I expected(!) your patch would do this to my output which would
break my mechanism. In fact, it didn't, but only because I have "+"
before the echo rules (for a different reason) which causes make to
consider them separate recipes (at least for the purpose of this
check). So, it's OK for me at the moment, but just coincidentally.
(I checked, if I remove the "+", it does just what I expected and
breaks my mechanism.)

Of course, you could say I'm misusing the same message channel for
different purposes here, but that's just the only channel readily
available without adding some message-passing framework or whatever.
(Sure, there are two channels, stdout and stderr, but separating the
messages between them wouldn't help here.)

> > > I think we're
> > > doing too much here.  I would have left it alone but I think some tools
> > > that parse make results expect to see that output to help them figure
> > > out what's going on.
> > 
> > As I described in https://savannah.gnu.org/bugs/?33138#comment3, the
> > problem is how to interpret messages that contain file names when
> > different jobs run in different subdirectories.
> 
> I do get it.  I just think we might need to do more drastic surgery on
> this to REALLY get it right.

I'm not saying it's perfect now, but my priorities are first correct
(i.e. sufficient context), then nice (i.e. no redundant messages).

> Probably we'll want to allow the user to
> have more control over it, as well.  Maybe a similar flag that lets you
> choose whether to trace on a per-target or per-make basis.

I think it should in principle be possible without requiring the
user to specify any more options. But it would be some work,
requiring make to keep track of which directory message was output
last, delay the "leaving" message in case the next one will be
"entering" the same directory etc., and synchronize this among
recursive makes in the different modes.

> And, if you
> choose per-target you don't need to ALSO generate output per-make.

Probably, though I didn't check it in detail. I just didn't want to
remove any messages that were output before.

> We do need to be careful about tracing non-target output, for example
> output generated by $(info ...), $(warning ...), and $(error ...).

The latter two are covered by the changes in error() and fatal(),
but $(info) was missing. At least in function.c, this seems to be
the only place that was forgotten:

--- function.c.orig     2013-04-18 11:39:15.000000000 +0200
+++ function.c  2013-04-18 20:08:35.000000000 +0200
@@ -1106,8 +1106,12 @@
       break;
 
     case 'i':
+      if (output_sync)
+        log_working_directory (1, 1);
       printf ("%s\n", msg);
       fflush(stdout);
+      if (output_sync)
+        log_working_directory (0, 1);
       break;
 
     default: