Re: Tokenizing

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    I suggest the following plan:

    1) try to implement the tokenizer using available text property mechanics;

    2) see if there are slow-downs or problems, or space for improvements
    on the Emacs side.

Yes.

At various points you might have ideas for more convenient or faster
predefined functions for accessing the text properties.  It will be
very nice to add them for Emacs and thus make this sort of tokenizing
more convenient.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.