Re: Reliable after-change-functions (via: Using incremental parsing in Emacs)

[Eli Zaretskii]

> From: Stephen Leake <address@hidden>
> Date: Wed, 01 Apr 2020 15:38:26 -0800
> 
> Eli Zaretskii <address@hidden> writes:
> 
> > Also, direct access to buffer text generally means we must make sure
> > GC never runs as long as pointers to buffer text are lying around.
> > Can any Lisp run between calls to the reader function that the
> > tree-sitter parser calls to access the buffer text?  
> 
> If the parser copies the text into an internal buffer, that reader
> function should only be called once per call to the parser.

Such copying is not really scalable, and IMO should be avoided.
During active editing, redisplay runs very frequently, and having to
copy portions of the buffer, let alone all of it, each time, which
necessarily requires memory allocation, consing of Lisp objects, etc.,
will produce significant memory pressure, expensive heap
allocations/deallocations, and a lot of GC.  Recall that on many
modern platforms Emacs doesn't really return memory to the system,
which means we risk increasing the memory footprint, and create
system-wide memory pressure.  It isn't a catastrophe, but we should
try to avoid it if possible.

> Since Emacs has the entire file in memory, the parser can too.

Having the file twice or more in memory is worse than having it only
once.

> However, if we are really trying to avoid copying text (which is very
> premature optimization)

I don't think it's premature.

> In sum, the short answer is "yes, you must parse the whole file, unless
> your language is particularly simple".

Funny, my conclusion from reading your detailed description was
entirely different.

> > IOW, the issue with exposing access to buffer text to modules is IMO
> > secondary.  
> 
> yes, because copying text is fast compared to everything else going on.

That wasn't my motivation when I wrote that.

> In general, each parser library, and even each grammar author, will have
> different representations for the syntax tree.
> 
> So if we want to support different parsers, I think it is best to define
> the Emacs "parser API" as "give text to parser; accept text properties
> from parser".

Yes, something like that.  It's probably enough to accept a list of
regions with syntactic attributes.