bug#31290: Fundamental bugs in syntax-propertize

[Alan Mackenzie]

Hello, Dmitry.

On Tue, May 08, 2018 at 15:35:14 +0300, Dmitry Gutov wrote:
> On 4/28/18 12:08 AM, Alan Mackenzie wrote:

> > At least that would be true if syntax-propertize--done hadn't been
> > prematurely and spuriously increased, crudely to prevent an infinite
> > recursion, falsely indicating to the syntax-ppss infrastructure that the
> > syntax-table properties have already been applied to the region (BEGIN
> > END).

> >      .... but it should not call `syntax-ppss-flush-cache', ....

> > Why not?  Because syntax-ppss-flush-cache sets syntax-propertize--done
> > back to its true value, allowing the wrongly allowed syntax-ppss calls at
> > a later position to cause a recursive loop.

> Maybe we should "allow" it to loop, in certain cases? Leaving it to be 
> the responsibility of the programmer, to make sure the result doesn't 
> infloop, even if these rules are violated.

I'm not sure how this could work.  We would need to formalise the rules
very carefully, to avoid the need to read syntax.{c,el}'s source code.

> >      .... which means that it should not call `syntax-ppss' on some
> >      position and later modify the buffer on some earlier position.

> > This is a bad restriction, because sometimes syntax-table properties can
> > only be correctly determined by examining the syntax of later buffer
> > positions.  An example of this is giving the string-fence syntax-table
> > text property to an unbalanced opening string quote, but not to correctly
> > matched quotes.

> I'm not exactly convinced by the given example (why would we use the 
> string-fence in that case?), but it might be better if something like 
> this was possible, indeed.

String fence can be used to signal to font lock that the delimiter
(together with the "mismatching" unescaped EOL) should be fontified in
warning face.

A better example might be C++ Mode's marking of a "< ... >" pair with
paren syntax.  This isn't done with syntax-propertize-function (as you
know), but it would be nice if this were possible.

> > 2. syntax-propertize-function's are banned from using syntax-ppss, the
> > documentation instead directing them to use parse-partial-sexp directly.

> The ones that currently call syntax-ppss, can't simply switch over to 
> parse-partial-sexp without becoming slower due to the lack of cache.

The cache at the pertinent buffer position doesn't exist at the time:
consistent syntax-table properties aren't on the preceding buffer
positions.

> Before tackling this bug, I'd rather we see a real-world problem that it 
> caused, and pick a particular approach based on it.

My enhancements for bug#30393: "24.4; cperl-mode: indentation failure -
Documentation enhancements", where (almost) any change which affects the
syntactic state is programmed to call syntax-ppss-flush-cache from the C
level, clashes with the mechanism in this bug report.  Most of the time
it's fine, but when a change affecting the syntactic state is made from
inside a synax-propertize-function, Emacs goes into an infinite recursive
loop.

This isn't good.

> But off the top of my head, we could introduce a "stricter but somewhat 
> slower" variation of syntax-ppss to be called inside 
> syntax-propertize-function's, which would treat the values in question 
> more carefully, somehow.

That's an idea worth exploring.

-- 
Alan Mackenzie (Nuremberg, Germany).