Re: call-process should not block process filters from running

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: sbaugh@catern.com
>> Date: Mon, 03 Jul 2023 16:28:25 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> > I'm not sure the new code should call swallow_events, even if
>> > protected by some variable.  We will be giving people a too-long rope
>> > to hang themselves.  It is unreasonable to assume Lisp programmers
>> > will be able to decide when to allow this and when not to, given the
>> > complexities.
>> 
>> What is complex about this?
>
> Whatever caused you to be surprised that just calling
> wait_reading_process_output doesn't process selection requests.  How
> many Lisp programmers understand what is going on in Emacs when such a
> request is processed, and what that means for their Lisp programs?
>
>> What problems can it cause that aren't already covered by "arbitrary
>> Lisp will run during call-process"?
>
> Processing selection requests does not really qualify as "running
> arbitrary Lisp".  It is much more subtle, and I'm quite sure the
> mental model of that which most Emacs users have is very simplistic.

I take your point, but can you be a little more concrete?  I genuinely
don't know what potential issues you're referring to that can be caused
by processing selection requests in a new place.  I'm asking just to
improve my knowledge of Emacs.

>> If swallow_events is too general, I can just call only
>> process_special_events if necessary.
>
> They are both potentially dangerous.
>
>> AFAIK this is a pretty small change, all it changes is letting Emacs
>> handle X selection requests while in call-process.  (i.e. letting other
>> X clients paste when Emacs owns the selection)
>
> Famous last words ;-)
>
> There are no "small changes" in Emacs on this level.
>
>> > Why do we need to jump through all the hoops right at the beginning?
>> > Why not make a small step forward and see if this experiment is more
>> > useful than it is dangerous?  Then we could decide whether to make the
>> > next step, and how.  No one said that pasting into Emacs while a
>> > subprocess runs is the most important capability to enable.  I'd first
>> > make sure the user can type at the keyboard without adverse effects.
>> 
>> It's not pasting into Emacs, it's pasting into other X clients.  I
>> indeed don't care about pasting into Emacs while a subprocess runs.  But
>> I like to be able to use other X clients, including by pasting, while a
>> subprocess runs.
>
> Still not important enough, IMO, for us to have to solve this right
> away, since you will have the same issue when Emacs is busy with
> something else, like redrawing the display or even executing some
> long-running Lisp program.

Redrawing the display is much quicker, and I don't use any long-running
Lisp programs.  In fact, I move long-running operations out into
asynchronous subprocesses so they can run concurrently with Emacs.  It's
only call-process usage that makes Emacs noticeably busy for me.

(As a tangent, it's kind of sad that this call-process API has ever
caused Emacs to block/stop running Lisp - subprocesses are inherently
concurrent!  They are what we use if we want Emacs to *not* block.  I
understand that that's complex, but it's just kind of ironic.)

>> > Let's not over-engineer this without a sufficient justification.
>> 
>> I've received lots of user complaints about the inability to paste in
>> other X clients while Emacs is sitting in call-process.  Fixing that is
>> one of my key motivations for this change.
>
> Nothing's wrong with fixing this step by step, from where I stand.

OK.  So what's the next step in this step-by-step from your perspective?
Should we install the option on trunk, let people use it, and gradually
make what it does broader and more fine-grained over time?