mmm-mode needs (was: narrow-to-here-document)

[Stefan Monnier]

> Our current implementation is able to do reasonably well by using
> font-lock-fontify-region-function and font-lock-beginning-of-syntax-function,
> but we'd love to have better support from font-lock (and other
> packages) for this sort of thing.  Thanks, Stefan, for crossposting to
> the mmm list, since I wasn't following emacs-devel.

I wish package developers were more proactive in requesting features.
Especially since oftentimes the feature already exists (although not
in your case, I'm afraid) but also because it gives us more information
about what direction Elisp should be heading into.

> Briefly, the way mmm-mode works is to parse the buffer for regions
> that it thinks (based on its configuration) ought to be edited using a
> different major mode, and create overlays to mark those positions.

Is the parsing done "eagerly" when opening a file, or is it done
on-demand, à la jit-lock ?  If done eagerly, has the parsing time
ever been a problem (as in font-lock where the solution was to have
font-lock-maximum-size before jit-lock) ?

What kind of granularity has mmm-mode been used at ?
I.e. has it been used with hundreds of "regions" ?
What about treating strings and comments as separate regions ?

> (It's not currently as good as we'd like at updating these overlays
> dynamically as text is edited, but I believe this is just because no
> one has sat down to write good code to do it.)

I tend to think of strings and comments as being in the same category
as here-documents, C-in-yacc, javascript-in-html, ...
For strings and comments, the boundaries are recognized by the syntax-tables
and syntax-ppss can be used to efficiently determine the "current mode"
as well as the starting point of that mode (and together with
parse-partial-sexp is can also give you the end point).  It doesn't use
overlays, but the dynamic updating works rather well.

What do you think about extending syntax-tables to recognize an extensible
set of "mode switch" markers (rather than just comment-start, comment-end,
and string).

> I don't mean to co-opt this disucssion for mmm, since it started on a
> somewhat different (though related) topic.  But a "wish list" of

I'm highly interested in Emacs's support for language syntax and mmm-mode
is an important aspect of it.

> 1. Have a "registry" of some sort for major modes, where they can
> store enough information for mmm to turn them on effectively in small
> regions.  Currently, the way mmm finds out what local variables and so
> on to set for each mode is to create temporary buffers and save the
> values of individual local variables.  Among other things, this
> requires mmm mode knowing about all local variables that any major
> mode could want to set, and so we have a huge list of what variables
> to look for and what modes to look for them in.  In addition, some

Why not use `buffer-local-variables' ?
[ Don't take that to mean I think you're an idiot or something:
  I'm just trying to better understand where the current features
  are failing. ]

> major modes expect things like a file name set in all buffers, so we
> have to do kludgy things to the temporary buffer to try to make them
> work.

Why do you have to use a temporary buffer rather than the current
buffer exactly ?

> 2. Have some sort of built-in support for variables that are localized
> to a text region within a buffer, such as an overlay or a text
> property.  I'm not sure how or if this would work in general, but it's
> effectively what mmm does already to some extent.

Is there a particular problem with the way mmm-mode does it ?

> In particular, some
> variables want to keep different values in different regions even of
> the same major mode, in particular variables that store the state of
> syntax parsing.  This leads to...

I don't understand this part.  Since currently such region-local variables
don't exist, packages use other ways to store the info than a variable:
either in a list (that's what I use in syntax-ppss) or in overlays or
text-properties, ...

> 3. Better support in the syntax parsing routines allowing the caller
> to specify starting and ending positions and states.  One of

That's generally between very difficult and impossible: many/most
indentation algorithms don't have an explicit notion of `state'
(they actually work backwards from the indentation-point trying to find
a place w.r.t which they can indent).

> our biggest problems is that submodes end up trying to parse their
> syntax beyond the beginning of their region and getting confused, so
> ideally we would be able to tell them where to start their parsing,
> and what state to start parsing in.  Sometimes the start state needs
> to be a continuation of the end state of the previous region in that
> mode (as with some literate programming, PHP, etc.), but sometimes
> it ought to be an independent start for each region.
> 
> This is similar to the 'syntactic realm' idea that David Kastrup
> suggested.  I don't think we've tried using intangible properties, for
> the reasons mentioned and perhaps others, but Joe Kelsey has had some
> success using syntax-table text properties to make the text that's not
> supposed to be parsed be treated as a single word or as whitespace, as
> appropriate.  Joe, want to comment?

I think making the "outside" text intangible or whitespace-like or word-like
or comment-like is indeed the best approach.  Fiddling with syntax-tables
is not 100% because many indentation algorithms don't use syntax-tables or
at least not exclusively, so they may not always expect to have to jump over
a comment, or they use "[ \t]" rather than "\\s-" or ...

Has anybody tried to play with point-entered/point-left or with
intangible ?

In any case it seems difficult with the current Emacs to make it work
reliably without having to adjust the indentation code in a few places,
so the Joe's approach might indeed be the best option (together with
a bit of collaboration from the major mode).

> 4. Effectively the same thing for font-lock, which uses the syntax
> parsing functions but also its own parsing functions.  In general
> we've had more success hacking around font-lock than around syntax,
> because font-lock provides more customization variables and its state
> is easier to understand.

Also font-lock doesn't keep much state between lines.
Is mmm-mode line-oriented as well ?

> If we want to continue this discussion, should it stay crossposted, or
> only on emacs-devel?

It should definitely be on emacs-devel.  Whether it should be crossposted
to mmm-mode, I don't have an opinion.


        Stefan