Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines

[Frank Heckenbach]

Eli Zaretskii wrote:

> > If Windows has a function to make a copy of a FD, whatever it is,
> > with new flags, this plus dup2 would be mostly equivalent to fcntl
> > for our purposes indeed.
> 
> On Windows, file descriptors are created and maintained by the C
> runtime, and they are private to the application.  Each descriptor is
> an index into an array which holds the underlying Windows handle for
> the file object and a bunch of flags, one of which is O_APPEND.  Those
> flags are used by the Posix emulation APIs, in this case 'write', to
> move the file pointer to the end of the file on each call to 'write'.
> The OS knows nothing about those flags, it manipulates the file using
> the handle and doesn't care about the descriptor.
>
> There's a library function to get a handle that corresponds to file
> descriptor, and another one that takes a handle and returns a new
> descriptor which references that handle.  The latter function accepts
> flags, including O_APPEND, to use for the file descriptor.
> 
> So the plan is:
> 
>   . get the handle that corresponds to (e.g.) stdout
> 
>   . produce a new descriptor for that handle with O_APPEND flag
> 
>   . use dup2 to replace the original stdout descriptor with this new
>     descriptor

I see, this might work.

However, there may still be a problem. The trick about O_APPEND on
POSIX it that it's atomic, i.e. nothing can get between moving the
file pointer and the write, even if another process tries to write
simultaneously. So if the POSIX emulation API emulates it with two
system calls (seek and write), it wouldn't be atomic. I don't know
if it does, or whether the problem exists in the first place. So for
all I know there are 3 possibilities:

a) Writing without O_APPEND is "synchronized" (see below) already.
   No need to do anything then.

b) Not a), and writing with O_APPEND is not synchronized either
   (perhaps because the emulation layer just does two separate
   system calls). Setting O_APPEND would be pointless then.

c) Not a), but O_APPEND is synchronized (probably because the
   emulation layer can use a system call to seek and write
   atomically). Only then it would make sense to do what you
   suggest.

> > (Though I doubt it has one, since from what I've seen, it generally
> > doesn't seem to treat files, pipes, etc.  uniformly.)
> 
> Windows does treat everything uniformly, just not the Posix way:

(I meant uniform WRT file operations. I remember how hard it was for
you to implement same_stream() with a solution that works one way
for files, another way for the console and not for the null device
etc. I heard about similar issues WRT "select", which is probably
also emulated and AIUI only works for sockets (and perhaps some
other devices), whereas on POSIX it accepts any FD. But we're
digressing, after your explanation above that's not the problem
here.)

> > > It sounds strange to me that the filesystem doesn't serialize the
> > > writes.  Maybe I'm naive.
> > 
> > I don't know the exact reasons. Perhaps it's just for efficiency, to
> > avoid synchronization by the OS for a rather special case, i.e.
> > different processes writing to the *same* file concurrently. If you
> > look at it this way, it smells like trouble because the question is,
> > how to merge the various writes.
> 
> I wasn't talking about synchronization or merging.  I was talking
> about _losing_ some of the output, which was the issue discussed here.

That's the consequence of lack of synchronization. As I understand
it, both writes take place, but at the same file position because
the offset is not updated before the 2nd write takes place. So they
overwrite each other and the 1st one gets lost.

> My interpretation of that is that the system writes to the file using
> more than a single file pointer.  And that is what sounded strange,
> because I always thought that a handle that was inherited from a
> parent process shares the same file pointer, and actually the whole
> underlying object used for the I/O, with the parent.  If that were
> true, then all the sub-makes would share the same file pointer, and we
> couldn't possibly lose any output due to overwriting.  Perhaps this
> information is outdated nowadays, though.

I think it's still true. It's just a question *when* the shared file
pointer is updated, i.e. there's apparently no critical section for
write plus seek. As I said, I don't know why this is so, maybe just
for efficiency since those who need it can work around (O_APPEND or
higher level synchronization).