Re: `make -O` when non-parallel

[Stefan Monnier]

Obviously, I did not look at the code to see how `make -O` is currently
implemented.  My (apparently incorrect) assumption was that it was
implemented by having a "master" process (the one that controls the
parallelism) be the only one that sends output to stdout: all the others
send their output (without delay) to the master which is in charge of
buffering and/or interleaving their outputs according to the
output mode.

In such a case, it seems the master could easily test if there's only one
active subprocess and if so send that subprocess's output directly to
stdout without having to buffer it until it ends.

IIUC this might require a non-trivial restructuring :-(

        Stefan


Paul Smith [2022-09-23 15:19:09] wrote:

> On Fri, 2022-09-23 at 13:23 -0400, Stefan Monnier via Users list for
> the GNU implementation of make wrote:
>> I like the fact that `make -O` gives me output that's always
>> meaningful, but it's sometimes annoying how it's sitting idly while a
>> command is spewing output.  To some extent it's unavoidable, but I
>> wonder if we could reduce the inconvenience by making such that Make
>> emits the subprocess's output without delay *if* that's the only
>> subprocess active (i.e. we're not waiting to see which process ends
>> first before emitting its output).
>
> I agree it's annoying but I don't quite understand the change you're
> suggesting.
>
> I guess you mean, some job would be given the real stdout FD rather
> than redirected to a temporary file, since that's the only way output
> could appear "without delay".
>
> Remember there's no way to answer "if that's the only subprocess
> active" in general, since we can have recursive make systems where many
> instances of make are running at the same time, all sharing the same
> stdout.
>
> In order to maintain ordered output, whichever make had invoked the
> "without delay" job would also have to hold the output lock for the
> duration of the run of that job (I mean, that's the only way it would
> be allowed to run "without delay").
>
> But the way output sync works is that a job is not considered "done"
> until its output is shoveled to stdout, and it waits to get the lock to
> do that.  The assumption is that no one job holds the lock for very
> long because they're just shoveling already-produced output to stdout.
> This means that if one job held that lock for a very long time, we
> would be waiting to complete (and start) other parallel jobs,
> effectively serializing the build.
>
> To allow this we'd have to break the "showing output" from the "running
> job" so we could complete jobs and start new ones, even though we
> hadn't shoveled the output to stdout yet, and keep the output temporary
> files in some a queue until we had a chance to write them out.  Which
> we could do but this could be a very large number of files left open.
>
> It's also not clear how to manage fairness: for a single instance of
> make it would be OK since after the "without delay" job finishes we
> could shovel all the already-done jobs before we started a new "without
> delay" job.  But if you had multiple instances of make it's not so
> simple.
>
> Remember, GNU make is not multithreaded, so it can only do one thing at
> a time and it doesn't have some kind of background event loop.
>
> One other thing: since the jobs that run "without delay" would have a
> real stdout, and the others would not, you might see different
> behaviors where some jobs printed colorized output etc. (because they
> detect they have a terminal) while others don't... I can imagine THAT
> would generate a lot of bug reports :).