Re: `make -O` when non-parallel

[Paul Smith]

On Fri, 2022-09-23 at 13:23 -0400, Stefan Monnier via Users list for
the GNU implementation of make wrote:
> I like the fact that `make -O` gives me output that's always
> meaningful, but it's sometimes annoying how it's sitting idly while a
> command is spewing output.  To some extent it's unavoidable, but I
> wonder if we could reduce the inconvenience by making such that Make
> emits the subprocess's output without delay *if* that's the only
> subprocess active (i.e. we're not waiting to see which process ends
> first before emitting its output).

I agree it's annoying but I don't quite understand the change you're
suggesting.

I guess you mean, some job would be given the real stdout FD rather
than redirected to a temporary file, since that's the only way output
could appear "without delay".

Remember there's no way to answer "if that's the only subprocess
active" in general, since we can have recursive make systems where many
instances of make are running at the same time, all sharing the same
stdout.

In order to maintain ordered output, whichever make had invoked the
"without delay" job would also have to hold the output lock for the
duration of the run of that job (I mean, that's the only way it would
be allowed to run "without delay").

But the way output sync works is that a job is not considered "done"
until its output is shoveled to stdout, and it waits to get the lock to
do that.  The assumption is that no one job holds the lock for very
long because they're just shoveling already-produced output to stdout.
This means that if one job held that lock for a very long time, we
would be waiting to complete (and start) other parallel jobs,
effectively serializing the build.

To allow this we'd have to break the "showing output" from the "running
job" so we could complete jobs and start new ones, even though we
hadn't shoveled the output to stdout yet, and keep the output temporary
files in some a queue until we had a chance to write them out.  Which
we could do but this could be a very large number of files left open.

It's also not clear how to manage fairness: for a single instance of
make it would be OK since after the "without delay" job finishes we
could shovel all the already-done jobs before we started a new "without
delay" job.  But if you had multiple instances of make it's not so
simple.

Remember, GNU make is not multithreaded, so it can only do one thing at
a time and it doesn't have some kind of background event loop.

One other thing: since the jobs that run "without delay" would have a
real stdout, and the others would not, you might see different
behaviors where some jobs printed colorized output etc. (because they
detect they have a terminal) while others don't... I can imagine THAT
would generate a lot of bug reports :).