CEDET, DL & parsing thoughts (was Re: Release plans)

[Eric M. Ludlam]

Hi,  Again no threading info, sorry.

>    Surely XRefactory's big advantage over CEDET is use of an EDG-based
>    parser (which costs money)? So in that sense the restrictions on how
>    the core gcc project develops (whether it can provide suitable dumps
>    of parse trees and the like) are more significant than restrictions on
>    Emacs?
>
>It could be.
>
>Hmm.  There is an architecture to consider:  Imagine dynamically
>loadable parsers for your favorite languages.   Might there be a
>reasonable API design such that a single parsing tool can do both
>incremental parsing / re-parsing and efficient straight-through parsing,
>producing output (in the form of API calls or return values) suitable
>for both building GCC trees and updating text properties and database
>values in an IDE?
>
>
>If so, can tools such as Bison be extended to support generation
>of the incremental (re-) parsing parts (e.g., with suitable ways of
>handling parse errors and recovery in an incremental context)?
>
>
>The resulting "kit" of Emacs w/DL + GCC w/DL +  extended Bison
>could be very fun to play with.

This is in effect what is in CEDET/Semantic now but without the DL.  I
had made a replacement for flex, but more Emacs Lisp centric, and
David Ponce ported bison into Emacs Lisp directly.  This bison port
supports incremental parsing, full parsing, reparsing, and is quite
fast, though not nearly as fast as actual flex/bison/c code.

I would assume the concepts in David Ponce's wisent parser generator
could be back-ported into Bison if desired.

If DLL's had existed before we did this, I would have liked to find a
way to feed an actual flex routine from a buffer, and have that feed
into a bison generated language.  Since those create fcns w/ a single
name, that could be hidden in the dll.  I also would have "borrowed"
the gcc .y file as a start.  I obviously didn't do this, nor did I try
the "subprocess", because I wanted "as you type" syntax checking,
which almost exists in the current version of CEDET. Doing that in an
external proces is irrational to do in an external program.  (See
flymake)

When I started, I really wanted to have a single generic parsing
infrastructure that could do indentation, coloring, and tagging.  As
it stands, I only really had time to focus on one thing, so I picked
that which had not been done, which is the dynamic tagging/completion
part.  This is the same state XRefactory is in.  The main difference,
however, is that XRefactory only does after-you-save tag management.
The integrated parser in CEDET will do as-you-type retagging, plus a
wide range of high-level decorating, and some powerful defun-level
movement, editing and folding.  As an out-of-Emacs process, XRefactory
has on-disk tables of tag usages which CEDET doesn't try to store in
Emacs process memory.

Unrelated to the DLL issue, one thing I think Emacs would benifit
from, is a single place where someone working on a "major-mode" could
encode the nature of the language.  Right now there are syntax-tables,
font-lock tables, imenu regexp, etags regexps, and, if you are lucky,
a robust indentation engine with some hairy partial-parsing in it.

I think it would be much nicer (which is why I've worked on it for so
long) to have a single "parser" that knows the language, that would
then be used by generic font lock, tagging, and indenting engines.  I
think the parser David Ponce made is a great place for tagging, and is
likely a great place to also embed the other parts, but it is likely
it will always be considered "slow" compared to the cute short-cuts
you find in font-lock and custom indentors.

Once CEDET is merged into Emacs, I hope to examine some of the speed
issues with others who know more what Emacs' internals are like.  (As
an FYI, all of CEDET's papers should now be in order for this.)

Thanks
Eric

-- 
          Eric Ludlam:                       address@hidden
   Siege: www.siege-engine.com          Emacs: http://cedet.sourceforge.net