Re: new module pipe-filter

[Paolo Bonzini]

On 07/20/2009 01:22 AM, Bruno Haible wrote:
> Eric Blake wrote:
> > > Such piping, where you write from the current process and read into
> > > the current process, may hang on BSD systems, because some data is
> > > present in system buffers but the system wants the buffers to be full
> > > before it continues. A fix for this hang is to enable non-blocking I/O
> > > and use a loop with select() that alternately reads and writes. See
> > > <http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=blob;f=gettext-tools/src/msgfilter.c;hb=HEAD#l580>
> > Any chance of porting that over to gnulib?
>
> I'm adding a new module 'pipe-filter', that does this.

I don't like very much the design of this module.  The problem is that 
it requires excessive effort on the child's part to buffer writes and/or 
to prepare all of the filter's input upfront.

A nicer interface IMHO could be:

/* Create a subprocess and pipe some data through it.
   progname is the program name used in error messages.
   prog_path is the file name of the program to invoke.
   prog_argv is a NULL terminated argument list, starting with
   prog_path as first element.
   If null_stderr is true, the subprocess' stderr will be redirected
   to /dev/null, and the usual error message to stderr will be
   omitted.  This is suitable when the subprocess does not fulfill an
   important task.
   If exit_on_error is true, any error will cause the main process to
   exit with an error status.
   If the subprocess does not start correctly, exit if exit_on_error is
   true, otherwise return NULL and set errno.

   The caller will write to the subprocess through filter_write; during
   calls to filter_write, the done_read function may be called to
   process any data that the subprocess has written.  done_read will
   receive at most read_bufsize bytes stored into buf, as well as a
   copy of private_data.  */

extern struct filter *
filter_create (const char *progname,
               const char *prog_path, char **prog_argv,
               bool null_stderr, bool exit_on_error,
               const char *buf, size_t read_bufsize,
               done_read_fn done_read,
               void *private_data);


/* Write size bytes starting at buf into the pipe and in the meanwhile
   possibly call the done_read function specified in create_filter.
   The done_read function may be called in a different thread than
   the current thread, depending on the platform.  However, it will
   always be called before filter_write has returned (or else will be
   delayed to the next call to filter_write or filter_close).  Return
   only after all the entire buffer has been written to the pipe.

   If there is a problem reading or writing, return -1 and set errno.
   If the subprocess exits early with nonzero status, return the status.
   (In either case, filter_write will instead exit if exit_on_error was
   passed as true).

   If the subprocess exits early with zero status, subsequent writes
   will becomes no-ops and zero is returned.

   Otherwise return 0.  */
extern int filter_write (struct filter *f,
                         const char *buf, size_t size);

/* Finish reading the output via the done_read function specified in
   create_filter.  The done_read function may be called in a different
   thread than.  However, it will always be called before filter_close
   has returned.  The write side of the pipe is closed as soon as
   filter_close starts, while the read side will be closed just before
   it finishes.  If there is a problem reading or closing the pipe,
   return -1 and set errno.  If the subprocess exits early with nonzero
   status, return the status.  (In either case, filter_close will
   instead exit if exit_on_error was passed as true).

   Otherwise return 0.  */
extern int filter_close (struct filter *f);

Paolo