[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Tokenizing

From: Eric Ludlam
Subject: Re: Tokenizing
Date: Sat, 20 Sep 2014 16:16:03 -0400
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.3a1pre) Gecko/20091222 Shredder/3.1a1pre

 On 09/20/2014 12:40 PM, Vladimir Kazanov wrote:
Tokenizing the whole buffer after any change is easily fast enough (on
modern hardware), even on a 7000 line buffer. Semantic parsing gets a
lot slower.

This is what I do right now in my prototype of a smarter Python mode.
The tokenizing process itself is usually fast enough. But parsing is
more complicated, and may take some time to rebuild the parse tree.
Incremental approach is a natural step here.

I caught only the tail of this thread, so I apologize if I refer to the incorrect thing.

A year ago or so we were talking about ada-mode, a modified parser, and how it might integrate with CEDET/Semantic on the CEDET mailing list. Is it still 'wisi', a different flavor of 'wisent' ?

If calls into your parser are being handled by parts of CEDET/Semantic for creating tags, then there is an incremental parser that you can enable that works with two of the other parser types included. The act of tagging the buffer enables it to track edits, and only reparse the bits that change. It handles incomplete code, and waits for it to be completed. There are many other possible states as well.

I certainly recommend trying to take advantage of that if the parsing you refer to is somehow related to Semantic. I'll be happy to help you figure out how to make your parser work in that framework.

If this thread isn't about that, then carry on... :)


reply via email to

[Prev in Thread] Current Thread [Next in Thread]