Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines

[Frank Heckenbach]

Eli Zaretskii wrote:

> > From: Frank Heckenbach <address@hidden>
> > 
> > However, there may still be a problem. The trick about O_APPEND on
> > POSIX it that it's atomic, i.e. nothing can get between moving the
> > file pointer and the write, even if another process tries to write
> > simultaneously. So if the POSIX emulation API emulates it with two
> > system calls (seek and write), it wouldn't be atomic. I don't know
> > if it does, or whether the problem exists in the first place.
> 
> The problem exists, but there's nothing that can be done about it, as
> long as we use write/fwrite/fprintf for this: the call to 'write'
> isn't atomic on Windows even without O_APPEND, because of the
> text-mode translation of newlines to CR-LF pairs.

I didn't mean the whole write() must be atomic, just the seek and
write part. I.e., if the translation is done in user-space and then
a system call does the write and seek atomically, it might still be
OK even without O_APPEND.

> > b) Not a), and writing with O_APPEND is not synchronized either
> >    (perhaps because the emulation layer just does two separate
> >    system calls). Setting O_APPEND would be pointless then.
> 
> No, it's not pointless.  It makes the problem smaller.  And if the
> Posix systems will do that, doing that on Windows will minimize the
> number of #ifdef's, of which we have way to many already.

As I undestood you, you'd have to write an emulation for
fcntl (F_SETFD). That code has to be written and maintained, and it
adds a (small) runtime overhead. Not sure if that's worth saving an
#ifdef, *if* the problem doesn't actually exist.

> > > > (Though I doubt it has one, since from what I've seen, it generally
> > > > doesn't seem to treat files, pipes, etc.  uniformly.)
> > > 
> > > Windows does treat everything uniformly, just not the Posix way:
> > 
> > I heard about similar issues WRT "select", which is probably
> > also emulated and AIUI only works for sockets (and perhaps some
> > other devices), whereas on POSIX it accepts any FD.
> 
> That's because a socket is not a file on Windows.

That's what I meant. On POSIX, stdout could be a socket. It might be
unusual for make (though I actually once had a web application that
ran make to produce various files on demand, however it returned the
files made, not make's output, but it was close ... ;-). But since
this is probably not possible on Windows (if a socket is not a file,
stdout can't be connected to one), we don't have to care about this
situation there.

> > > I wasn't talking about synchronization or merging.  I was talking
> > > about _losing_ some of the output, which was the issue discussed here.
> > 
> > That's the consequence of lack of synchronization.
> 
> No, it isn't.  If the same file pointer were used, there would be no
> need for any synchronization, because that pointer would serialize
> output by its very nature.

Not sure what you mean here. The OS is not a magic box that does
anything "by its very nature". AIUI, it may contain code roughly
like this:

void write (int fd, void *data, size_t size)
{
  if (getflags (fd) & O_APPEND)
    {
      lock_mutex (get_mutex (fd));
      off_t pos = get_size (fd);
      do_write (fd, pos, data, size);
      set_pos (fd, pos + size);
      unlock_mutex (get_mutex (fd));
    }
  else
    {
      // no mutex here!
      off_t pos = get_pos (fd);
      do_write (fd, pos, data, size);
      set_pos (fd, pos + size);
    }
}

Different code paths, different behaviour.