Re: Reliable after-change-functions (via: Using incremental parsing in Emacs)

[Tuấn-Anh Nguyễn]

On Thu, Apr 2, 2020 at 10:02 PM Eli Zaretskii <address@hidden> wrote:
>
> > From: Tuấn-Anh Nguyễn <address@hidden>
> > Date: Thu, 2 Apr 2020 11:21:49 +0700
> > Cc: address@hidden
> >
> > > Buffer text is not exactly UTF-8, it's a superset of UTF-8.  So one
> > > question to answer is what to do with byte sequences that are not
> > > valid UTF-8.  Any suggestions or ideas?  How does tree-sitter handle
> > > invalid byte sequences in general?
> > >
> >
> > I haven't checked yet. It will probably bail out, which is usually the
> > desired behavior.
>
> "Bail out" meaning that this breaks the parse?  I'd be surprised if
> that was what happens in these cases.  But if it does, we will need to
> replace such sequences by the likes of U+FFFD in the reader function
> we provide.
>

Agreed. I'll try checking its behavior on this.

> > > IOW, the issue with exposing access to buffer text to modules is IMO
> > > secondary.  My suggestion is first to figure out how to do this stuff
> > > efficiently from within Emacs itself, as if the module interface were
> > > not part of the equation.  We can add that aspect back later.
> > >
> >
> > My opinion is that it's better to experiment with this kind of stuff
> > out-of-core. It can move forward faster that way, allowing more lessons
> > to be learned. Real lessons, involving real-world use cases, not thought
> > exercises.
>
> I'm talking about trying different design ideas.  It is best to do
> that without being limited by what modules can and cannot do.
> Building a hacked version of Emacs to test those ideas doesn't
> necessarily contradict the desire to collect real-life experience.
>
> IOW, I suggest to test alternative design ideas that are not based on
> copying portions of the buffer via Lisp strings.  If those ideas are
> workable (and I think they are), they will support a more scalable
> implementation that exerts less memory pressure on Emacs and on the
> host system.
>
> HTH
>

Yeah, I agree that going through Lisp strings for this is sub-optimal.
When I have time to come back to this part, I'll hack up my local Emacs
to allow dynamic modules to access buffer texts directly, to test out
the idea.

--
Tuấn-Anh Nguyễn
Software Engineer

P.S. Sorry Gmail messed up my first reply.