Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines

[Eli Zaretskii]

> Date: Thu, 30 May 2013 04:18:22 +0200
> Cc: address@hidden, address@hidden, address@hidden
> From: Frank Heckenbach <address@hidden>
> 
> >   . get the handle that corresponds to (e.g.) stdout
> > 
> >   . produce a new descriptor for that handle with O_APPEND flag
> > 
> >   . use dup2 to replace the original stdout descriptor with this new
> >     descriptor
> 
> I see, this might work.
> 
> However, there may still be a problem. The trick about O_APPEND on
> POSIX it that it's atomic, i.e. nothing can get between moving the
> file pointer and the write, even if another process tries to write
> simultaneously. So if the POSIX emulation API emulates it with two
> system calls (seek and write), it wouldn't be atomic. I don't know
> if it does, or whether the problem exists in the first place.

The problem exists, but there's nothing that can be done about it, as
long as we use write/fwrite/fprintf for this: the call to 'write'
isn't atomic on Windows even without O_APPEND, because of the
text-mode translation of newlines to CR-LF pairs.

> b) Not a), and writing with O_APPEND is not synchronized either
>    (perhaps because the emulation layer just does two separate
>    system calls). Setting O_APPEND would be pointless then.

No, it's not pointless.  It makes the problem smaller.  And if the
Posix systems will do that, doing that on Windows will minimize the
number of #ifdef's, of which we have way to many already.

> > > (Though I doubt it has one, since from what I've seen, it generally
> > > doesn't seem to treat files, pipes, etc.  uniformly.)
> > 
> > Windows does treat everything uniformly, just not the Posix way:
> 
> (I meant uniform WRT file operations. I remember how hard it was for
> you to implement same_stream() with a solution that works one way
> for files, another way for the console and not for the null device
> etc.

It was not a file operation that was a problem, it was the Posix-only
concept of inodes.  Emulating a paradigm from another OS is always
hard.  I'm sure the same would happen to Posix code if it were to try
emulating native Windows ops.

> I heard about similar issues WRT "select", which is probably
> also emulated and AIUI only works for sockets (and perhaps some
> other devices), whereas on POSIX it accepts any FD.

That's because a socket is not a file on Windows.

> > I wasn't talking about synchronization or merging.  I was talking
> > about _losing_ some of the output, which was the issue discussed here.
> 
> That's the consequence of lack of synchronization.

No, it isn't.  If the same file pointer were used, there would be no
need for any synchronization, because that pointer would serialize
output by its very nature.

> I think it's still true. It's just a question *when* the shared file
> pointer is updated, i.e. there's apparently no critical section for
> write plus seek.

If there's only one pointer, this is not an issue.