Re: locked narrowing in ELisp

[Dmitry Gutov]

On 19.08.2022 09:31, Eli Zaretskii wrote:

> > The redisplay could repeat the trick I showed with Isearch and wrap most
> > of its logic in (with-soft-narrow ...) -- which translates the soft
> > narrowing into an actual one temporarily. Then the 'invisible' property
> > won't even be required.
>
> I think you still underestimate the magnitude of the problem and its
> fundamental nature.  The display engine is not a closed subsystem, it
> uses a huge number of Emacs infrastructure code from other subsystems:
> buffers, text properties, overlays, markers, character composition,
> regular expressions, direct access to buffer text, etc.  All of those
> places are also used from other commands and functions, outside of
> redisplay.  You cannot "wrap [their] logic in (with-soft-narrow ...)",
> because their logic is used in many more use cases, and there's no way
> such low-level code can distinguish between the cases, nor should it
> (because it will make it too slow).

Naturally, some uses of 'goto-char' (for example) need to obey the 
narrowing applied by the user, and some -- do not. That's the situation 
today, and that's not going to change.

The question is, would "do not obey" work as a good default.

> > > That would mean all of the commands.  A lot of changes (or a lot of
> > > breakage).  It doesn't sound like a good solution, at least not better
> > > than just biting the bullet and introducing 2 different kinds of
> > > narrowing with some ways of telling Emacs internals which of the two
> > > to obey at any particular moment.
> >
> > Two kinds of narrowings is inherently a more complex solution.
>
> Some problems don't have simple solutions.  The narrowing is a very
> fundamental feature in Emacs, it permeates all of our code to the
> lowest levels.  Such fundamental features cannot be side-stepped by
> tricks and hacks.  The change must be in those same basic levels.

I think we're talking about two very similar things here, just looking 
at them from different angles.

But if we were going to separate user-level narrowing from "real" 
narrowing, the result should look (at least at the surface) like the 
patch I posted, and it should face the same problems: how to decide 
which commands should obey user-level locking, and how to minimize the 
number of functions we'll have to alter to support it explicitly.

> > > (The latter is actually the tricky
> > > part, IMO.)
> >
> > If we just make the "locking" feature to be accessible to Lisp, the
> > majority of the code shouldn't care or be aware of it. Just some
> > particular features/functions/packages will make use of the locking
> > argument, limiting the 'widen' calls in the rest of the Emacs from
> > widening past the locked region.
>
> I'm not talking about locking!  I'm talking about the basic
> requirement to distinguish between "user-level" narrowing and the
> other kind.  How do you tell the low-level stuff, such as
> set_marker_restricted, which of the two narrowing kinds to apply?

The solution I picked is this: lower-level code only obeys the "real" 
narrowing. Perhaps the redisplay loop sprinkles in a bit of support for 
"user-level" narrowing by temporarily applying the "real" narrowing 
around the whole main logic when such "user-level" narrowing is present 
(using its bounds).

As a result, with very rare exceptions (Info-mode, maybe), the only kind 
of "persistent" narrowing would be the "user-level" one, but every 
command (or function, or form) will have the freedom to translate it 
into the "real" one for the duration of its execution. The "real" 
narrowing would be thus relegated to lower-level uses, and not as a 
user's tool.

> That function is used from every place where we set window-point and
> window-start positions (as well as many others), so almost any code
> that wants to move one of those will eventually call it.  Commands do
> that, but redisplay and other places do it as well.
>
> This is the tricky part of introducing two kinds of narrowing; the
> rest is very simple and basically mechanical.
>
> > > Making a specific command ignore narrowing is easy.  Your proposal
> > > implicitly assumes that the number of commands that want to ignore
> > > narrowing is much larger than the other kind.
> >
> > That is indeed the key assumption. I'm inclined to believe that not only
> > it is larger, but that the set of commands that should ignore user-level
> > narrowing also grows faster.
>
> What about commands that apply narrowing as part of their internal
> operation -- is that "user-level" narrowing or the other kind?  E.g.,
> some/many commands that operate on the region when it's active begin
> by narrowing to that region -- which kind is that?

It's the other kind, the "real" one.

Sometime later, new functions might crop up (probably in third-party 
code) that decide to alter the user-level narrowing temporarily instead 
because they know which of their callees obey user-level narrowing and 
which do not, and decide to benefit from that distinction. But that 
approach should remain rare.

> > With that in mind, creating any precise statistics doesn't seem
> > possible, even if one decides to try.
>
> If that's so, then any assumptions about which set is significantly
> larger cannot be safely made, either.  We should proceed without any
> such assumption, i.e. without adopting any solutions which would be
> based on such assumptions and will break if the assumptions are proved
> false.

We could try to make such analysis anyway, just remain aware that the 
difference might be larger (in one particular direction).

If we just have to adopt a solution which goes against the assumption I 
made, well, then, the approach probably won't work.

> > On the flip side, though, it doesn't seem like diff-hl needs any support
> > for user-level narrowing (ignoring it is fine), and company-mode "just
> > worked" when I tested it with soft-narrow-to-region.
> >
> > Same goes for all other packages I maintain.
>
> IME, it is not a good idea to base such fundamental design decisions
> on a couple of user commands and their needs.

Several fairly high-profile packages are not "a couple of user 
commands". But it's not the whole ecosystem, sure.

> > > Moreover,
> > > I think it might make sense for some commands to honor or ignore the
> > > narrowing depending on the use case, and that is not solved with your
> > > proposal.
> >
> > Doesn't seem difficult:
> >
> > (if something
> >       (with-soft-narrow
> >        do-the-thing)
> >     do-the-thing)
> >
> > or without the macro:
> >
> > (save-restriction
> >     (when something
> >       (let ((bounds (soft-narrow-bounds)))
> >         (and bounds
> >              (narrow-to-region (car bounds) (cdr bounds)))))
> >     do-the-thing)
>
> We are mis-communicating: the problem is not how to use basic
> conditionals in Emacs Lisp, the problem is how you design and
> implement that "something", such that we won't need to rewrite every
> single command out there.

I considered which commands should be affected the most: they have to be 
those that act on and affect the part of the buffer that the user sees. 
The user shouldn't be able to modify the text they don't see (1), nor 
(probably) have that text affect the commands they are calling (2).

The (1) is implemented by adding the 'read-only' property to the 
overlays which apply the user-level narrowing. It leaves the 
possibilities of the user being able to jump before the first overlay 
(to bob) or after the last, and insert something there. That seems to be 
mostly handled using the FRONT-ADVANCE and REAR-ADVANCE arguments, as 
well as modification-hooks. But there are other options like 
post-command-hook, or making the overlays longer than 0 chars.

As for (2), though, we can't tell that in advance for each command, and 
most have to choose case-by-case. But Stefan gave a good example of when 
we *do* know: when the command does something with the region. The user 
cannot tell the region spans beyond the accessible/visible part of the 
buffer.

So the way to "fix" a lot of them together is to make sure the region 
cannot span beyond the user-level narrowing currently applied. How to do 
that? Either we have to change all of the commands which change the 
region, or alter 'set-mark' (which might be a fairly invasive change, 
stopping certain code from working), or perform the adjustments in 
post-command-hook: after every command, see if the region spans beyond 
the user-level narrowing, and adjust either or both of its bounds if needed.