[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: call-process should not block process filters from running

From: Spencer Baugh
Subject: Re: call-process should not block process filters from running
Date: Sat, 01 Jul 2023 15:17:39 -0400
User-agent: Gnus/5.13 (Gnus v5.13)

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Sat, 01 Jul 2023 14:24:59 -0400
>> >> And AFAIU, such a way does exist.  The implementation of call-process
>> >> actually forks a subprocess, then reads from its pipe until EOF, and
>> >> then waits in waitpid for the subprocess to exit.  Both the reading
>> >> and the waiting are done in a loop.  So one way of making call-process
>> >> less blocking is by adding to these loops calls to
>> >> wait_reading_process_output, like we do, for example, in sleep-for,
>> >> but conditioned by some new variable exposed to Lisp.  Lisp programs
>> >> which want this behavior could then activate it by binding that new
>> >> variable.  This could give us most or all of the advantages of
>> >> non-blocking API without most disadvantages.  (We'd still need to move
>> >> to this functionality on a case by case basis, because some of the
>> >> considerations against that could still be valid in individual cases.)
>> >
>> > This sounds great, I would be happy to implement this.  I think we
>> > would also want a tiny wrapper in Lisp which binds this new variable
>> > then calls call-process, rather than having lots of programs binding
>> > the variable directly, to make it easier to change the implementation
>> > strategy in the future.
>> >
>> > I'll work on implementing this new variable.  If you have any other
>> > suggestions for it, let me know.
>> Just to sanity check before I go down the wrong path: When this variable
>> is set, instead of doing the reading from the subprocess's pipe in
>> call-process, we'll need to do it in wait_reading_process_output, so
>> that other Lisp can run.  We can't just add in calls to
>> wait_reading_process_output alongside the existing calls to read,
>> because read is blocking, and we need to process other input if there is
>> some, even if there's no input from the process under call_process.  A
>> similar change will need to happen for waitpid handling.
> We read from pipe in chunks, and my idea was to leave the reading code
> intact, just call wait_reading_process_output each iteration through
> the reading loop.  Long-running call-process calls spend their time in
> this loop, reading the process output that takes long time to produce.
> So that's where I envision most of the gains will come from.

That's certainly better than nothing, but that won't run other Lisp
while the call-process process is not producing output.  Just as one
degenerate example, call-process on "sleep 30" won't run any Lisp.  More
realistically, expensive computations which take a while to produce
output ("factor $BIGNUM" for example) will not run much Lisp.  From
looking at call-process usage, I think most are of that form.  So I
don't think this will solve the problem.

> As for waitpid, I'd initially leave that alone, and only add the
> wait_reading_process_output to the loop that protects that from
> signals.  IME, by the time we get to waitpid after reading the last
> portion of the process output, the process have either already exited
> or will do so almost instantly.  If we discover this is not so, we
> could later replace waitpid with some wait with timeout in a loop.

I see.  But if that's the case, then that's not really any better than
just calling reading input and running other Lisp after that loop is
done, which of course we already do.  Because, if waitpid returns
immediately anyway, it doesn't really matter if we run Lisp before it or
run Lisp after it.

> I didn't mean to throw away the reading loop and waitpid, because
> doing so will require to write a lot of non-trivial C code, and goes
> against what I consider to be the main advantage of my idea.

Yes, I understand that you don't want to introduce bugs into
call-process.  IMO that's why making a completely new function instead
of modifying call-process is a better bet.

But if you would prefer a modification to call-process I'm quite happy
to write non-trivial C code, as long as the problem actually gets

reply via email to

[Prev in Thread] Current Thread [Next in Thread]