Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines

[Paul Smith]

On Mon, 2013-05-27 at 20:13 +0300, Eli Zaretskii wrote:
> > and if so, does it guarantee non-conflicting writes?
> 
> Not sure I understand what you are asking here.  Can you elaborate?

The original issue reported is that if you do something like this:

    make -j >make.out

and your make environment is recursive so you invoke one or more
sub-makes, your output may not just be interspersed (that is output
between multiple jobs are mixed together) but you will actually lose
some output: it will never appear at all.

The reason is that when you have multiple processes trying to update the
same file at the same time using standard output file mode, there is a
race condition between when the output is written to the file and when
the "current offset" value is updated, where multiple processes could be
overwriting the same part of the file.

The suggested solution (not modifying make) is to use this instead:

  : >make.out # truncate the file
  make -j >> make.out

POSIX guarantees that if you open a file in O_APPEND mode, the above
race can never happen because the kernel updates the file offset as the
file is being written.

Frank's question is whether other, non-POSIX systems have the same
behavior with O_APPEND.  Of course if they don't I don't see how it
would make things worse than they are now.

What I was suggesting was having make itself reset the mode of stdout
and stderr to add O_APPEND, so that the first (most common) syntax would
work correctly.  POSIX says that you can change the mode of an open file
descriptor using fcntl().

This wouldn't hurt anything in the above case, because when the shell
opens the output file (with O_TRUNC) it will be truncated, then it will
give the FD to make and make will change the mode to O_APPEND, so the
file will still be truncated as you expect.

The only possible way this could burn someone is if they are invoking
make from a program where they've specifically opened make's
stdout/stderr without O_APPEND and without O_TRUNC, and they expect make
to start overwriting the file from the beginning rather than appending
to the end.  I cannot conceive of any situation where something like
that would be done intentionally.