[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Make run in parallel mode with output redirected to a regular file c

From: Frank Heckenbach
Subject: Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines
Date: Fri, 31 May 2013 16:58:21 +0200

Eli Zaretskii wrote:

> > From: Frank Heckenbach <address@hidden>
> > 
> > Eli Zaretskii wrote:
> > 
> > > The problem exists, but there's nothing that can be done about it, as
> > > long as we use write/fwrite/fprintf for this: the call to 'write'
> > > isn't atomic on Windows even without O_APPEND, because of the
> > > text-mode translation of newlines to CR-LF pairs.
> > 
> > I didn't mean the whole write() must be atomic, just the seek and
> > write part. I.e., if the translation is done in user-space and then
> > a system call does the write and seek atomically, it might still be
> > OK even without O_APPEND.
> But that's what I'm telling you: each chunk of text after NL to CR-LF
> conversion is written separately in a separate call to WriteFile,
> which is the low-level API for file I/O.

Actually you hadn't told me this before. I had assumed the data was
converted as a whole and then written in one go. Now what you say
makes sense to me.

So the original problem (losing some output) still might not exist,
but a different problem might exist (unintended line-wise mixup of
different outputs) and might not be fixed by O_APPEND.

> > As I undestood you, you'd have to write an emulation for
> > fcntl (F_SETFD). That code has to be written and maintained, and it
> > adds a (small) runtime overhead. Not sure if that's worth saving an
> > #ifdef, *if* the problem doesn't actually exist.
> Given your next message about tmpfile, I will need that anyway.

It's the same situation (just seen from the other end), so if you
don't need O_APPEND for stdout/stderr, you won't need it for the
tmpfiles either.

> > > > > I wasn't talking about synchronization or merging.  I was talking
> > > > > about _losing_ some of the output, which was the issue discussed here.
> > > > 
> > > > That's the consequence of lack of synchronization.
> > > 
> > > No, it isn't.  If the same file pointer were used, there would be no
> > > need for any synchronization, because that pointer would serialize
> > > output by its very nature.
> > 
> > Not sure what you mean here. The OS is not a magic box that does
> > anything "by its very nature".
> No magic needed when there's only one file pointer, because a single
> file pointer can only be at one place at any given time.

Yes, but not necessarily at different places at different times:

> > void write (int fd, void *data, size_t size)
> > {
> >   if (getflags (fd) & O_APPEND)
> >     {
> >       lock_mutex (get_mutex (fd));
> >       off_t pos = get_size (fd);
> >       do_write (fd, pos, data, size);
> >       set_pos (fd, pos + size);
> >       unlock_mutex (get_mutex (fd));
> >     }
> >   else
> >     {
> >       // no mutex here!
> >       off_t pos = get_pos (fd);
> >       do_write (fd, pos, data, size);
> >       set_pos (fd, pos + size);
> >     }
> > }
> If the 'else' clause uses a single file pointer system-wise, there's
> no overwriting because the pointer is not moved between writes.

I still can't follow you. Just imagine this function is run by two
different processes simultaneously with the same FD without
O_APPEND. Both fetch the current position (get_pos) and get the same
value. Then both write (do_write) at this same position, overwriting
each other. Finally, both update the file pointer (set_pos), but
again, only the 2nd one becomes effective.

That's a rather typical lack-of-synchronization situation, and a
mutex around the code would fix it, because one process wouldn't be
able to get_pos before the other one has finished and done set_pos.
If I was designing a system, I might have done it this way, but fact
is POSIX doesn't mandate it, so we can't assume it. But perhaps
Windows does so, and then this problem doesn't exist there.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]