Re: [Emacs-diffs] scratch/widen-less a4ba846: Replace prog-widen with consolidating widen calls

[Dmitry Gutov]

On 12/3/17 5:20 PM, Eli Zaretskii wrote:

> > This feature is *not* targeted at Emacs development.
>
> It is for me.

You are not working on it. You've expressed no interest in this problem 
until now. Maybe delegate the basic requirements, at least, to the 
people who did?

> > > Indeed; and C-like languages are quite popular among them.
> >
> > Not in comparison to JS, CSS and HTML.
>
> Maybe in your world, but not in mine.  So we will have to agree (or
> disagree, if you want) that both worlds need to be catered to.

Yes, and they can be catered to, *as soon as* the authors of each 
individual major mode make the effort to support the feature.

Having it work in HTML, CSS, JS and Ruby is in no way dependent in 
investigating and fixing the behemoth that CC Mode is.

> > > How will we look if we support JS and Ruby embedded in HTML, but not C
> > > code embedded in Yacc grammars nor Awk snippets embedded in shell
> > > scripts?  Both are quite frequent out there.  We will be laughed at.
> >
> > We'll be fine.
>
>  From your POV, maybe.  Not from mine.

Eli, we already have a feature in Emacs (prog-indentation-context) that 
does not adhere to your (unreasonably high) requirements.

Let's at least take it out, then.

> > > Convince Alan to do what?
> >
> > To adhere to the current proposal (avoid widening in
> > indent-line-function and font-lock-keywords, to start with).
>
> We need to understand the full extent of problems, and then we need to
> see how to go about them, at least design-wise.  "To start with" is a
> start, but it cannot be the middle and the end.

The full extent of the problem can be realized as one tried to do what I 
said, step by step. And see what else can be is going wrong.

> If he agrees, that'd be swell.  If not, I see no reason why someone
> else couldn't do that and report back, it's not like CC Mode is hard
> to activate and see what problems pop up.

Please go ahead and do it, then.

If you're really expecting me to do it, it's unrealistic. Getting 
oriented in CC Mode will take days, if not weeks. And I have no interest 
nor energy to do that.

> > I have a rough understanding of the issue, but since I haven't reached a
> > working state, I don't know how many pitfalls there are left.
>
> How about describing what you do know?

Already did. Last time I tried investigating it was 1-2 years ago, and 
at no point I felt I reached an understanding of why CC Mode exhibits 
the problems it does, or what I could change to fix them.

So all I have are vague recollections.

> > I imagine the process itself might be trickier than expected. Various
> > primitives use caches that save context information. What is such
> > primitive to do if the cache contains "beginning of nesting" outside of
> > the current restriction, and the logic of said primitive says "go to the
> > beginning of the current function and do such and such"? The answer
> > isn't obvious to me.
>
> I don't understand: AFAIU in the use cases supported by MMM there can
> be nothing outside of the current restriction that is of interest for
> the mode which supports the chunk under the restriction.  So what kind
> of "nesting outside of the current restriction" are we talking about,
> and how does it come into play?

Imagine if the chunk contains only this:

   <

And, to fontify it, CC Mode, tried to find out what syntactic element it 
is, and to do that, calls some function that uses a specialized cache 
that's contained in a variable that mmm-mode knows nothing about.

And that cache point to a position outside of the current restriction. 
Confusion ensues. I've seen errors originating from that (I think), but 
can't recall the exact sequence of calls.

> > The proposal itself is very small, and there's not much to explain. Just
> > look at the changes in the manual, in this branch.
>
> That just removes text, more or less, and what it adds doesn't explain
> itself well enough, sorry.  Otherwise I wouldn't have asked these
> questions, believe me.

The text it adds is more important. Including the mentions of 
font-lock-keywords and syntax-propertize-function, by the way.

> > It simply facilitates what mmm-mode (and other modes, including
> > mhtml-mode) has been doing for years, with varying success.
>
> Sorry, that doesn't help me understand the proposal, as I don't know
> mmm-mode well enough.  How hard is it to just describe the idea in a
> few sentences?

Not really sure what to describe.

To fontify (or indent) a chunk, mmm-mode narrows to its bounds, and 
calls the corresponding indent-line-function (or applies 
font-lock-keywords). If either of those calls (widen), that causes problems.

So we want to document a convention that they don't call widen. That's 
really it.

> > I never actually supported it, just stopped arguing because I didn't
> > have a good alternative idea. Now I do, I think. And there is some
> > agreement from Stefan.
>
> (whose change of hearts is also a mystery for me)

Sometimes a better technical proposal does that.

For more context: we're been waiting for some kind of improvement for 
this problem for years. Then prog-indentation-context came along, and it 
wasn't terrible (though it didn't solve all of the problems we wanted). 
That seemed a good enough reason to adopt it, to some of us.

Since then, for the mentioned two years, the original author contributed 
no improvements aside from the mentioned support in python-mode, and no 
other mode added support for it, or used it, inside or outside Emacs. So 
it seems to me like it pretty much failed.

> > > How do you address the issues which prog-indentation-context did
> > > (e.g., if the embedded chunk of code is incomplete, and perhaps even
> > > syntactically invalid)?
> >
> > prog-indentation-context never addressed that issue, and we don't
> > either.
>
> I think it does, at least to some extent.

Not any more than scratch/widen-less.

> And even if I'm wrong, then
> what is the equivalent of what it does address in your proposal?

There are full equivalents.

We keep 'prog-first-column' from that proposal, and instead of allowing 
MMM to indicate the chunk bounds through prog-indentation-context, we 
allow MMM to apply narrowing directly, and that modes honor it.

> In any case, we should at least have provisions for supporting that
> within the proposed framework, and we should be fairly certain that
> supporting that later will not blow things up.

I have honestly no idea where to begin in addressing that issue.

On the other hand, there are third-party MMM framework that function 
within these limitations, and are helpful to users.

> > That's the thing: we're not giving up much
>
> For the modes that are dear to your heart, maybe.  But that's not all
> Emacs should support well, IMO.

For all modes. The user won't lose any functionality, at all. Even if 
the programmer might have to spend some effort for that to happen, for 
some of the modes.

> The 'widen' thing is a non-issue; it's the other stuff I'm worried
> about, mostly because it is left unsaid, undiscussed, and I fear
> uncatered-to by the design.

What other stuff? We basically say "widen happens in one place, don't do 
that again", and that's it.

> > Not really. There's a minor issue of whether to make prog-first-column a
> > variable, or a hook right away, but the importance of that choice isn't big.
>
> The fact is, it isn't done yet, let alone well-tested.

It can be "done" in a day or two.

> > > That code is in Emacs for more than 2 years.  It was admitted with
> > > Stefan's full support, and at least ANTLR needs it in conjunction with
> > > Python.
> >
> > Transitioning from prog-indentation-context to the new approach is very
> > easy.
>
> That's what I thought.  And that's why we should do that
> simultaneously with landing the new approach.  There's no reason to do
> that earlier.

Why confuse the language modes authors?

> > So really, it's been here for 2 years, and virtually nobody's using it,
> > or improving it.
>
> It's definitely used by its author, so "nobody uses it" is untrue, and
> even unfair: that feature was accepted by Stefan, under the assumption
> that it will be used.

...and improved, and support for CC Mode would be contributed. Which it 
wasn't.

The author, who's the only user of prog-indentation-context, can 
transition to the new convention trivially.

> > > Removing it without having some alternative again makes no
> > > sense to me.
> >
> > You have the alternative, though.
>
> No, we don't.  You are asking to remove the code _before_ the
> alternative lands.

You have the choice of accepting the alternative sooner, though.

> > > We should discuss this when the incompatible code lands;
> >
> > How about we discuss it now?
>
> The incompatible code didn't land yet.  But feel free to include the
> replacement in your branch.

It's in there already.

> > Take a look at any third-party packages. I'm willing to bet none use
> > prog-indentation-context yet.
>
> We cannot know that.  Once the code is that long with us, we cannot
> throw it away without a equivalent replacement.  And I see no reason
> to do that now, since the fact that this code will exist in Emacs 26
> doesn't hurt anyone, especially since the replacement will be easy, as
> you say.

Well... you have a point there. The result will be pretty ugly, though. 
Adding an API in one version, and removing it in another.

They are incompatible with each other, so it's not like there can be a 
smooth transition.