Re: patch to support output synchronization under -j

[David Boyce]

On Fri, Apr 15, 2011 at 9:48 AM, Eli Zaretskii <address@hidden> wrote:
> So effectively, whenever a job finishes, its parent Make takes the
> semaphore, outputs all of the output of that job to the screen, then
> releases the semaphore, is that right?

Yes, exactly.

> And how do you communicate the name of the semaphore file to
> sub-Make's?  Is that by assuming that its file descriptor is
> inherited?
>
> I also don't understand the need in the MAKESYNCFILE variable: why is
> it important where the semaphore file will live and how will it be
> named?

Addressed by second iteration. There is no sync file anymore.

> Finally, wouldn't it be a potential problem top inherit so many
> handles to subordinate processes (2 for each running job)?  We could
> run out of available handles in deeply recursive jobs, couldn't we?

Just to be completely clear, we're not talking about a change to
default behavior - it's merely a new option. But yes, any feature
which consumes additional resources increases the risk of running out
of that resource. I don't know that recursion comes into it though,
it's more a question of how many parallel jobs a given make process is
managing since limits on file handles/descriptors are per process. In
other words if using the jobserver you're limited to the number of
jobs specified at the top level even in a recursive setup. If not,
make tries to work around that by forcing -j1, and if that doesn't
work you're in trouble anyway.

The strictest file limit I know of is on old Unixy platforms. The
tmpfile() API returns a FILE *, not a file descriptor, and older
Unices have a limit of 256 of these. But even in that corner case, how
many people are going to run make -j200+? How many 64-CPU processors
are running really old kernels? So yes, while there is a theoretical
limit there are many kinds of resource exhaustion which can be
triggered by large values of -j and as far as I can tell this is just
another, and optional.

Parenthetically, I only use FILE * because that's what tmpfile hands
back. I believe it would be possible to dup the file descriptor out of
the FILE *, close it right away, and work with an integral file
descriptor from there. That would avoid the above limit.

David