Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines

[Eli Zaretskii]

> Date: Wed, 29 May 2013 09:32:55 +0200
> Cc: address@hidden, address@hidden
> From: Frank Heckenbach <address@hidden>
> 
> Eli Zaretskii wrote:
> 
> > > Date: Mon, 27 May 2013 00:42:34 +0200
> > > From: Frank Heckenbach <address@hidden>
> > > Cc: address@hidden
> > > 
> > > One issue, though it might seem strange that I'm the one to mention
> > > it, is that it might be POSIX specific. How do other systems behave,
> > > can they set O_APPEND via fcntl or otherwise
> > 
> > This can be done on Windows by creating a new file descriptor that has
> > the O_APPEND bit set, and then using dup2 to force stdout/stderr refer
> > to that file descriptor.  (This is theory; I should try that and see
> > if it actually works.)
> 
> I don't think this would work, as least on systems I know (mostly
> POSIX)

I was talking specifically about Windows, because that's what your
question above was about.

> since we're talking about altering the flags of the
> stdout/stderr given to us. We don't usually have its filename to
> open it again; it may not even have a filename (e.g., it might be a
> file created and deleted; or it might be a pipe, a socket, etc.), or
> it might not be possible to reopen it (maybe we don't have
> permissions anymore; or again sockets) ...

I don't need the name of the file, all I need is its file descriptor
or its Windows handle.

> If Windows has a function to make a copy of a FD, whatever it is,
> with new flags, this plus dup2 would be mostly equivalent to fcntl
> for our purposes indeed.

On Windows, file descriptors are created and maintained by the C
runtime, and they are private to the application.  Each descriptor is
an index into an array which holds the underlying Windows handle for
the file object and a bunch of flags, one of which is O_APPEND.  Those
flags are used by the Posix emulation APIs, in this case 'write', to
move the file pointer to the end of the file on each call to 'write'.
The OS knows nothing about those flags, it manipulates the file using
the handle and doesn't care about the descriptor.

There's a library function to get a handle that corresponds to file
descriptor, and another one that takes a handle and returns a new
descriptor which references that handle.  The latter function accepts
flags, including O_APPEND, to use for the file descriptor.

So the plan is:

  . get the handle that corresponds to (e.g.) stdout

  . produce a new descriptor for that handle with O_APPEND flag

  . use dup2 to replace the original stdout descriptor with this new
    descriptor

> (Though I doubt it has one, since from what I've seen, it generally
> doesn't seem to treat files, pipes, etc.  uniformly.)

Windows does treat everything uniformly, just not the Posix way: every
object is referenced by a handle, which is an opaque pointer.  That
paradigm is actually broader than the Posix file descriptor paradigm:
there are objects, like events, critical sections, semaphores,
processes, etc. that are all referenced by handles, and there are APIs
that will take just about any handle and do their thing on it.  The
simplest example is CloseHandle.

> > It sounds strange to me that the filesystem doesn't serialize the
> > writes.  Maybe I'm naive.
> 
> I don't know the exact reasons. Perhaps it's just for efficiency, to
> avoid synchronization by the OS for a rather special case, i.e.
> different processes writing to the *same* file concurrently. If you
> look at it this way, it smells like trouble because the question is,
> how to merge the various writes.

I wasn't talking about synchronization or merging.  I was talking
about _losing_ some of the output, which was the issue discussed here.
My interpretation of that is that the system writes to the file using
more than a single file pointer.  And that is what sounded strange,
because I always thought that a handle that was inherited from a
parent process shares the same file pointer, and actually the whole
underlying object used for the I/O, with the parent.  If that were
true, then all the sub-makes would share the same file pointer, and we
couldn't possibly lose any output due to overwriting.  Perhaps this
information is outdated nowadays, though.